OpenGL & Kinect - Rendering ColorImageFrame Bitmap to OpenGL Texture

Mar 12, 2012 at 4:17 PM

So I want to draw a simple 2D quad with a texture from the Kinect's video input but my code isn't working. I'm not sure what I'm doing wrong...


Here is my ColorFrameReady event:

 

void Sensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
{
       ColorImageFrame cif = e.OpenColorImageFrame();

       if (cif != null)
       {
           middleBitmapSource = cif.ToBitmapSource();
           cif.Dispose();
       }
}

 

My OpenGL Draw method:

 

private void OpenGLControl_OpenGLDraw(object sender, OpenGLEventArgs args)
        {
            // Get the OpenGL instance being pushed to us.
            gl = args.OpenGL;

            // Clear the colour and depth buffers.
            gl.Clear(OpenGL.GL_COLOR_BUFFER_BIT | OpenGL.GL_DEPTH_BUFFER_BIT);

            // Enable textures
            gl.Enable(OpenGL.GL_TEXTURE_2D);

            // Reset the modelview matrix.
            gl.LoadIdentity();

            // Load textures
            if (middleBitmapSource != null)
            {
                middleBitmap = getBitmap(middleBitmapSource, quadMiddleWidth, quadMiddleHeight);
                //middleBitmap = new System.Drawing.Bitmap("C:\\Users\\Bobble\\Pictures\\Crate.bmp"); // When I use this texture, it works fine
                gl.GenTextures(1, textures);
                gl.BindTexture(OpenGL.GL_TEXTURE_2D, textures[0]);
                gl.TexImage2D(OpenGL.GL_TEXTURE_2D, 0, 3, middleBitmap.Width, middleBitmap.Height, 0, OpenGL.GL_BGR, OpenGL.GL_UNSIGNED_BYTE,
                    middleBitmap.LockBits(new System.Drawing.Rectangle(0, 0, middleBitmap.Width, middleBitmap.Height),
                    System.Drawing.Imaging.ImageLockMode.ReadOnly, System.Drawing.Imaging.PixelFormat.Format24bppRgb).Scan0);

                gl.TexParameter(OpenGL.GL_TEXTURE_2D, OpenGL.GL_TEXTURE_MIN_FILTER, OpenGL.GL_LINEAR);
                gl.TexParameter(OpenGL.GL_TEXTURE_2D, OpenGL.GL_TEXTURE_MAG_FILTER, OpenGL.GL_LINEAR);
            }

            // Move back a bit
            gl.Translate(0.0f, 0.0f, -25.0f);

            gl.BindTexture(OpenGL.GL_TEXTURE_2D, textures[0]);
            
            // Draw a quad.
            gl.Begin(OpenGL.GL_QUADS);

                // Draw centre quad
                gl.TexCoord(0.0f, 0.0f); gl.Vertex(-(quadMiddleWidth / 2), quadMiddleHeight / 2, quadDepthFar);
                gl.TexCoord(0.0f, 1.0f); gl.Vertex(quadMiddleWidth / 2, quadMiddleHeight / 2, quadDepthFar);
                gl.TexCoord(1.0f, 1.0f); gl.Vertex(quadMiddleWidth / 2, -(quadMiddleHeight / 2), quadDepthFar);
                gl.TexCoord(1.0f, 0.0f); gl.Vertex(-(quadMiddleWidth / 2), -(quadMiddleHeight / 2), quadDepthFar);

            gl.End();

            gl.Flush();
        }

 

And a helper method which gets the bitmap from a bitmap source

 

private System.Drawing.Bitmap getBitmap(BitmapSource bitmapSource, double rectangleWidth, double rectangleHeight)
        {
            double newWidthRatio = rectangleWidth / (double)bitmapSource.PixelWidth;
            double newHeightRatio = ((rectangleWidth * bitmapSource.PixelHeight) / (double)bitmapSource.PixelWidth) / (double)bitmapSource.PixelHeight;

            BitmapSource transformedBitmapSource = new TransformedBitmap(bitmapSource, new ScaleTransform(newWidthRatio, newHeightRatio));

            int width = transformedBitmapSource.PixelWidth;
            int height = transformedBitmapSource.PixelHeight;
            //int stride = width * ((transformedBitmapSource.Format.BitsPerPixel + 7) / 8);
            int stride = ((width * transformedBitmapSource.Format.BitsPerPixel + 31) & ~31) / 8; // See: http://stackoverflow.com/questions/1983781/why-does-bitmapsource-create-throw-an-argumentexception/1983886#1983886

            byte[] bits = new byte[height * stride];

            transformedBitmapSource.CopyPixels(bits, stride, 0);

            unsafe
            {
                fixed (byte* pBits = bits)
                {
                    IntPtr ptr = new IntPtr(pBits);

                    System.Drawing.Bitmap bitmap = new System.Drawing.Bitmap(
                        width, height, stride, System.Drawing.Imaging.PixelFormat.Format32bppArgb, ptr);

                    return bitmap;
                }
            }
        }

The quad renders but it is just white, no texture on it. Any help/advice would be appreciated.
Thanks in advance.


Link to my SO question in case anybody wants some virtual points. I'll post any good answers in this discussion too.

http://stackoverflow.com/questions/9669353/opengl-kinect-rendering-colorimageframe-bitmap-to-opengl-texture

 

Coordinator
Mar 14, 2012 at 2:58 PM

Just as a quick thought, have you tried using the SharpGL.SceneGraph.Assets.Texture object? This'll let you load the data without having to do any conversion. Then you can just call texture.Bind(gl). It may be worth a try - let me know how it goes

Mar 15, 2012 at 1:04 PM
Edited Mar 15, 2012 at 1:07 PM

Hi Dwmkerr. Thanks for your response.

Could you please give me an example of using SharpGL.SceneGraph.Assets.Texture. I can't see how I can use it in the context of Bitmap objects. You can assign a file path of a bitmap to a Texture object but not a Bitmap object from what I can see.

Thanks.

Coordinator
Mar 26, 2012 at 10:43 AM

Let me look into this for you - if you could send over the code I'll work out what's happening

Mar 28, 2012 at 6:04 PM
Edited Mar 29, 2012 at 10:26 AM

Hi dwmkerr, thanks for taking a look at this. I've been tinkering with the Texture object and it makes life a lot easier for texturing something from a single Bitmap saved in the file system. However, using the Kinect means I need to change textures every time there is a new video frame from the Kinect, so ~30 times a second. I'm not an expert with OpenGL but I'm confident that what I have just described isn't too demanding for it?

Converting the video frame to a Bitmap is no problem at all but the problem is that SharpGL's Texture object only accepts a path to a Bitmap saved within the file system. I've experimented with saving the video frame to a Bitmap file and then taking the texture from there but doing it that way crashes the program.

Edit: Did some research and basically I need to find a way to load the texture without it touching the hard disk as that will be a huge bottleneck and probably kill the disk.

Here is my code so far:

 

/// <summary>
/// Invokes whenever there is a new frame from the Kinect camera.
/// </summary>
void Sensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
{
	using (ColorImageFrame cif = e.OpenColorImageFrame())
	{
		if (cif != null)
		{
			// Save the frame to a Bitmap Source object.
			middleViewBitmapSource = cif.ToBitmapSource();

			// Update the source of the System.Windows.Controls.Image to that of the new frame.
                        // This isn't important, just outputs the video frame so we can see that it's working.
			imageVideoControl.Source = middleViewBitmapSource;

			// Convert the BitmapSource object to a Bitmap object.
			middleViewBitmap = BitmapConverter.BitmapFromSource(middleViewBitmapSource);

			// Save the Bitmap object to a file of type .bmp .
			middleViewBitmap.Save("Media/Textures/middleViewBitmap", System.Drawing.Imaging.ImageFormat.Bmp);
			
			// Dispose of the frame.
			cif.Dispose();
		}
	}
}
private void OpenGLControl_OpenGLDraw(object sender, OpenGLEventArgs args)
{
	// Get the OpenGL instance being pushed to us.
	gl = args.OpenGL;

	// Clear the colour and depth buffers.
	gl.Clear(OpenGL.GL_COLOR_BUFFER_BIT | OpenGL.GL_DEPTH_BUFFER_BIT);

	// Reset the modelview matrix.
	gl.LoadIdentity();

	// Move back a bit
	gl.Translate(0.0f, 0.0f, -25.0f);

	// Enable textures
	gl.Enable(OpenGL.GL_TEXTURE_2D);

	if (File.Exists("Media/Textures/middleViewBitmap"))
	{
		// If the bitmap file exists for the middle view, create and bind it to the Texture object.
		texMiddleView.Create(gl, "Media/Textures/middleViewBitmap");
		texMiddleView.Bind(gl);
	}
	else
	{
		// Else bind a crate texture to show something has gone wrong.
		texCrate.Bind(gl); // This texture is created in the OpenGL_Initialized method.
	}

	// TODO: Do the same for the left and right views.

	// Draw quad representing the middle view.
	gl.Begin(OpenGL.GL_QUADS);

		gl.TexCoord(0.0f, 0.0f); gl.Vertex(middleViewTopRight.X, middleViewTopRight.Y, middleViewTopRight.Z);
		gl.TexCoord(0.0f, 1.0f); gl.Vertex(middleViewTopLeft.X, middleViewTopLeft.Y, middleViewTopLeft.Z);
		gl.TexCoord(1.0f, 1.0f); gl.Vertex(middleViewBottomLeft.X, middleViewBottomLeft.Y, middleViewBottomLeft.Z);
		gl.TexCoord(1.0f, 0.0f); gl.Vertex(middleViewBottomRight.X, middleViewBottomRight.Y, middleViewBottomRight.Z);

	gl.End();

	texCrate.Bind(gl); // For now, just focus on middle view, left and right views can have a crate texture.

	// Draw left and right views.
	gl.Begin(OpenGL.GL_QUADS);

		gl.TexCoord(0.0f, 0.0f); gl.Vertex(leftViewTopLeft.X, leftViewTopLeft.Y, leftViewTopLeft.Z);
		gl.TexCoord(0.0f, 1.0f); gl.Vertex(middleViewTopLeft.X, middleViewTopLeft.Y, middleViewTopLeft.Z);
		gl.TexCoord(1.0f, 1.0f); gl.Vertex(middleViewBottomLeft.X, middleViewBottomLeft.Y, middleViewBottomLeft.Z);
		gl.TexCoord(1.0f, 0.0f); gl.Vertex(leftViewBottomLeft.X, leftViewBottomLeft.Y, leftViewBottomLeft.Z);

		gl.TexCoord(0.0f, 0.0f); gl.Vertex(middleViewTopRight.X, middleViewTopRight.Y, middleViewTopRight.Z);
		gl.TexCoord(0.0f, 1.0f); gl.Vertex(rightViewTopRight.X, rightViewTopRight.Y, rightViewTopRight.Z);
		gl.TexCoord(1.0f, 1.0f); gl.Vertex(rightViewBottomRight.X, rightViewBottomRight.Y, rightViewBottomRight.Z);
		gl.TexCoord(1.0f, 0.0f); gl.Vertex(middleViewBottomRight.X, middleViewBottomRight.Y, middleViewBottomRight.Z);

	gl.End();

	// Bind texture for the 'next' button.
	texBtnNext.Bind(gl);

	// Draw 'next' button.
	gl.Begin(OpenGL.GL_TRIANGLES);

		gl.TexCoord(0.0f, 0.0f); gl.Vertex(nextBtnTopLeft.X, nextBtnTopLeft.Y, nextBtnTopLeft.Z);
		gl.TexCoord(1.0f, 0.0f); gl.Vertex(nextBtnTopRight.X, nextBtnTopRight.Y, nextBtnTopRight.Z);
		gl.TexCoord(1.0f, 1.0f); gl.Vertex(nextBtnBottomRight.X, nextBtnBottomRight.Y, nextBtnBottomRight.Z);

	gl.End();

	// Bind texture for 'previous' button.
	texBtnPrev.Bind(gl);

	// Draw 'previous' button.
	gl.Begin(OpenGL.GL_TRIANGLES);
		
		gl.TexCoord(0.0f, 0.0f); gl.Vertex(prevBtnTopLeft.X, prevBtnTopLeft.Y, prevBtnTopLeft.Z);
		gl.TexCoord(1.0f, 0.0f); gl.Vertex(prevBtnTopRight.X, prevBtnTopRight.Y, prevBtnTopRight.Z);
		gl.TexCoord(0.0f, 1.0f); gl.Vertex(prevBtnBottomLeft.X, prevBtnBottomLeft.Y, prevBtnBottomLeft.Z);
		
	gl.End();

	// Disable textures.
	gl.Disable(OpenGL.GL_TEXTURE_2D);

	gl.LoadIdentity();

	/*************************************************************
	* Other stuff here for skeleton tracking, model drawing etc. *
	**************************************************************/

	texRightView.Destroy(gl);
	texMiddleView.Destroy(gl);
	texLeftView.Destroy(gl);

	gl.Flush();

	cursorRotate += 20.0f;
}
Coordinator
Mar 30, 2012 at 1:42 PM

Hi Bobble,

It looks to me like you've found a fairly hefty flaw in SharpGL - you should be able to create a SharpGL Texture object from a bitmap, not just a path. I'm going to raise this as a work item now, I'll update the system for you and we can take it from there :)

Coordinator
Mar 30, 2012 at 1:43 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Coordinator
Mar 30, 2012 at 1:43 PM

In case you want to follow it, the work item is here:

http://sharpgl.codeplex.com/workitem/701

Mar 30, 2012 at 2:56 PM
Edited Mar 30, 2012 at 2:57 PM

Thanks dwmkerr.

I spent a while yesterday working on this and produced some code which works for me which I have included below. Maybe it will help:

 

private void drawMiddleView()
{
	// If there is a frame in the middleViewBitmap object (i.e. if there is a current frame) then 
	// use this as a bitmap for texturing. 
	if(middleViewBitmapSource != null)
	{
		// Convert the bitmap source (the current frame) to a bitmap of a particular size.
		middleViewBitmap = BitmapUtilities.bitmapFromSourceWithSize(middleViewBitmapSource, 512, 512);

		// Bind the texture.
		gl.BindTexture(OpenGL.GL_TEXTURE_2D, textures[0]);

		// Define a new rectangle of the size of the bitmap.
		Rectangle rect = new Rectangle(0, 0, middleViewBitmap.Width, middleViewBitmap.Height);

		// Get a byte array of the bits of the bitmap.
		byte[] bits = BitmapUtilities.byteArrayFromBitmapSource(middleViewBitmapSource, (double)rect.Width, (double)rect.Height);

		// Tell OpenGL to create a TexImage2D with various parameters.
		gl.TexImage2D(OpenGL.GL_TEXTURE_2D,
					  0,
					  (int)OpenGL.GL_RGBA,
					  512,
					  512,
					  0,
					  OpenGL.GL_BGRA,
					  OpenGL.GL_UNSIGNED_BYTE,
					  bits);

		// Set texture 2D parameters
		gl.TexParameter(OpenGL.GL_TEXTURE_2D, OpenGL.GL_TEXTURE_MIN_FILTER, OpenGL.GL_NEAREST);
		gl.TexParameter(OpenGL.GL_TEXTURE_2D, OpenGL.GL_TEXTURE_MAG_FILTER, OpenGL.GL_NEAREST);
	} else
	{
		// If there is no frame from the camera, bind a static texture from file.
		texCrate.Bind(gl);
	}

	// Draw quad representing the middle view.
	gl.Begin(OpenGL.GL_QUADS);
		gl.TexCoord(0.0f, 1.0f); gl.Vertex(middleViewBottomLeft.X, middleViewBottomLeft.Y, middleViewBottomLeft.Z);
		gl.TexCoord(0.0f, 0.0f); gl.Vertex(middleViewTopLeft.X, middleViewTopLeft.Y, middleViewTopLeft.Z);
		gl.TexCoord(1.0f, 0.0f); gl.Vertex(middleViewTopRight.X, middleViewTopRight.Y, middleViewTopRight.Z);
		gl.TexCoord(1.0f, 1.0f); gl.Vertex(middleViewBottomRight.X, middleViewBottomRight.Y, middleViewBottomRight.Z);
	gl.End();
}

 

 

/// <summary>
/// BitmapUtilties class provides various static methods for converting System.Drawing.Bitmap objects.
/// </summary>
class BitmapUtilities
{
	/// <summary>
	/// Converts a BitmapSource object to a Bitmap object.
	/// 
	/// Source: http://stackoverflow.com/questions/3751715/convert-bitmapsource-to-image
	/// </summary>
	/// <param name="bitmapsource">The System.Windows.Media.Imaging.BitmapSource object to be converted.</param>
	/// <returns>A new System.Drawing.Bitmap object.</returns>
	public static Bitmap bitmapFromSource(BitmapSource bitmapsource)
	{
		Bitmap bitmap;
		using (MemoryStream outStream = new MemoryStream())
		{
			BitmapEncoder enc = new BmpBitmapEncoder();
			enc.Frames.Add(BitmapFrame.Create(bitmapsource));
			enc.Save(outStream);
			bitmap = new Bitmap(outStream);
		}
		return bitmap;
	}

	public static byte[] byteArrayFromBitmapSource(BitmapSource bitmapSource, double rectangleWidth, double rectangleHeight)
	{
		byte[] temp;
		double newWidthRatio = rectangleWidth / (double)bitmapSource.PixelWidth;
		//double newHeightRatio = ((rectangleWidth * bitmapSource.PixelHeight) / (double)bitmapSource.PixelWidth) / (double)bitmapSource.PixelHeight;
		double newHeightRatio = rectangleHeight / (double)bitmapSource.PixelHeight;

		BitmapSource transformedBitmapSource = new TransformedBitmap(bitmapSource, new System.Windows.Media.ScaleTransform(newWidthRatio, newHeightRatio));

		int width = transformedBitmapSource.PixelWidth;
		int height = transformedBitmapSource.PixelHeight;
		int stride = width * ((transformedBitmapSource.Format.BitsPerPixel + 7) / 8);

		temp = new byte[height * stride];

		transformedBitmapSource.CopyPixels(temp, stride, 0);

		return temp;
	}
}