Detecting image update changes (pixel changes)? to send over TCP/IP

Question

Clawsy 1 Posting Whiz in Training

14 Years Ago

Hello,

I have to continously update an image based on screen capture. So how can I memorize only the part of the image that changes from one picture to another (from one frame to another)(pixels?)... Think of a remote desktop. It's not a good thing to send a 200K desktop screenshot image to the client every time because you get a BIG delay and you get to low frames per second. Ideal is to send only the part of the image that changes.

I hope you understood what I an trying to do... So how can I detect the image changes between to frames? I can't loop through all pixel to see what pixel changed because it takes a BIG amount of time... I was wondering how , for example, TeamViewer has such a good refresh rate. I think it sends only the pixel information that changes from frame to frame. That's what I need so much.

Thanks.

image

4 Contributors
10 Replies
2K Views
9 Months Discussion Span
Latest Post 14 Years Ago Latest Post by mesho100200

sknake 1,622 Senior Poster

14 Years Ago

>> Dr_Gonzo: great post! I would argue the point of UDP being faster but regardless of the reason, it is the right choice for this job.

You want to use UDP for streaming content. Any time you don't care about data fidelity it is likely a candidate for UDP. Think of RDP -- if the connection drops out for a few seconds you don't want to replay all of the "video" you missed, you just want to see the current screen capture. That being said you should use UDP.

Regarding how to capture changes in the image -- I don't know this for certain but I bet you can hook in to the win32 API and catch portions of the screen that are repainted/invalidated, then update your in-memory copy with those regions. When you detect a region has been repainted send the position & size of the invalidated region, and update your client.

Like I said I don't know for certain but that is how I would approach it. Doing a pixel-by-pixel comparison to detect changes is computationally expensive and there has to be a better way.

sknake 1,622 Senior Poster

14 Years Ago

Take a look at this thread:
http://social.msdn.microsoft.com/forums/en-US/csharplanguage/thread/694b0ed0-802b-4724-ab2a-21b27421d49d

You'll need to set hooks and start capturing when windows begins painting invalidated regions. That region is your "changed" portion of the bitmap.

Reply to this topic

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.

Dr_Gonzo 0 Newbie Poster · Answer 1 · 2010-01-19T00:12:52+00:00

It would be pretty hard to detect what has changed since the last update, since the mouse could have been moved somewhere else, or a window could have been opened/closed/resized.

But the concept of 'updating only what has changed', is a something that is often applied in movie codecs (I and P frames). So perhaps that's something you could look into (perhaps through a UDP stream?).

Another option is to use a heavy (lossy) compression technique (JPEG?) to get smaller files. It uses principles like run length encoding that instead of 'red red red red' writes '4 red' to file. But the purpose of monitoring is that you see what happens, so perhaps a lossless alternative is worth checking out (PNG?).

E: Something else, what if you 'partition' your screen (in 4 or 8, 16 smaller images), and have your software scan if the diagonals of each partition are identical to the ones captured before? And then only send the partitions that were changed + their x and y position, to recreate the image at the other side. One caveat however, is to always evaluate the partition where the mouse pointer is located, as that might be too small for the 'detection' system to notice.

Although I don't know if the diagonals are 'enough' to be responsive enough, perhaps other patterns are worth looking into.

Also, why the requirement of TCP/IP? That requires a 3 way handshake to set up, and while you have more reliability, UDP is faster, but some packets may never arrive. But if the update frequency is high enough, I wonder if it matters that much.

Clawsy 1 Posting Whiz in Training · Answer 2 · 2010-01-19T06:18:17+00:00

Thanks for replies... I am weak in win32 API... I don't know if somebody can give me an example of approach to what Dr_Gonzo and sknake adviced me.. I would really appreciate it. I always enjoy learning more and more... I just need the headstart. Thanks.

Dr_Gonzo 0 Newbie Poster · Answer 3 · 2010-01-19T13:23:40+00:00

Thanks for replies... I am weak in win32 API... I don't know if somebody can give me an example of approach to what Dr_Gonzo and sknake adviced me.. I would really appreciate it. I always enjoy learning more and more... I just need the headstart. Thanks.

I've done some more research, I think the keywords here are 'dirty rectangles' as explained here.

There they also refer to the function getUpdateRgn, which might be something that could get you started.

Guide to WIN32 Paint for beginners. The links at the bottom of that page also seem useful. No idea whether GDI is also useful, as AFAIK it's not that fast.

Clawsy 1 Posting Whiz in Training · Answer 4 · 2010-01-22T11:59:14+00:00

Thanks for guiding. Very usefull. I read all you adviced. Now I think I would need at least how to call Win32 API from C# in order to get the coordonates of the Rectangle that will be repainted. So I need to call in fact GetUpdateRect() function from C# but I don't know how. This is API reference: http://msdn.microsoft.com/en-us/library/aa923766.aspx

//1) when I import this, gives it says RECT type not found... 
        [DllImport("user32.dll")]
        public static extern int GetUpdateRect(int hwnd, [MarshalAs(UnmanagedType.Struct)] ref RECT lpRect, int bErase);
//2. The second parameter would be the coordonates of the rectangle.. how can I get them?

Thanks in advance!

Clawsy 1 Posting Whiz in Training · Answer 5 · 2010-01-22T12:50:41+00:00

Sorry for posting twice.. here's what I found (still have a problem):
I have a class usefull for user32 functions. I use that class to get the region... I should get it but I always get 0,0,0,0 rectangle coordonates. WHY? Where's the mistake?

Here's the code for the program:

//WIN32 is the namespace for the classes I use
//PlatformInvokeUSER32 holds the WIN32 API functions
 //create the handle for the desktop
            IntPtr handle = PlatformInvokeUSER32.GetDC(WIN32.PlatformInvokeUSER32.GetDesktopWindow());
            //new rect structure to hold the coordinates
            rect = new RECT();
            //get the rectangle the encloses the last paint region -> why the coords are always 0 ???
            PlatformInvokeUSER32.GetUpdateRect(handle, ref rect, false);

            lbl_region.Text = "x1=" + rect.left.ToString() +
                " y1=" + rect.top.ToString() +
                " x2=" + rect.right.ToString() +
                " y2=" + rect.bottom.ToString();

PlatformInvokeUSER32 class:

/// <summary>
	/// This class shall keep the User32 APIs being used in 
	/// our program.
	/// </summary>
	public class PlatformInvokeUSER32
	{

		#region Class Variables
		public  const int SM_CXSCREEN=0;
		public  const int SM_CYSCREEN=1;
		#endregion		
		
		#region Class Functions
		[DllImport("user32.dll", EntryPoint="GetDesktopWindow")]
		public static extern IntPtr GetDesktopWindow();

		[DllImport("user32.dll",EntryPoint="GetDC")]
		public static extern IntPtr GetDC(IntPtr ptr);

		[DllImport("user32.dll",EntryPoint="GetSystemMetrics")]
		public static extern int GetSystemMetrics(int abc);

		[DllImport("user32.dll",EntryPoint="GetWindowDC")]
		public static extern IntPtr GetWindowDC(Int32 ptr);

		[DllImport("user32.dll",EntryPoint="ReleaseDC")]
		public static extern IntPtr ReleaseDC(IntPtr hWnd,IntPtr hDc);

        [DllImport("User32.dll",EntryPoint="GetWindowDC", CharSet = CharSet.Auto)]
        public static extern int GetUpdateRect(IntPtr hwnd, ref RECT rect, bool erase);

		#endregion

		#region Public Constructor
		public PlatformInvokeUSER32()
		{
			// 
			// TODO: Add constructor logic here
			//
		}
		#endregion
	}

	//This structure shall be used to keep the size of the screen.
	public struct SIZE
	{
		public int cx;
		public int cy;
	}

    public struct RECT
    {
        public int left;
        public int top;
        public int right;
        public int bottom;
    }

Why I get 0,0,0,0 coordonates for GetUpdateRect()? I called this code inside a button handler and also inside LocationChanged functions...
Please help!

Clawsy 1 Posting Whiz in Training · Answer 6 · 2010-01-22T18:35:53+00:00

Please... how can I obtain the area (rectangle) of the screen that was changed? I need this so much... Someone?

Clawsy 1 Posting Whiz in Training · Answer 7 · 2010-01-25T13:05:12+00:00

So I guess it's bad :(.... can't hook into WM_PAINT messages... that's what I'm trying to do. And if I would... why he says is very expensive? Would I have a big time delay to process this? Can you help me writing the code for hooking? Maybe my code doesn't work properly because it's not hooked exactly in WM_PAINED message. I'm in c#, I guess inc++ would be simplier to get into the message but I need this in c#

mesho100200 0 Newbie Poster · Answer 8 · 2010-11-16T06:11:20+00:00

here the big project do all

http://blog.bobcravens.com/2009/04/create-a-remote-desktop-viewer-using-c-and-wcf/