C / C++ / MFC

6-Jul-12 0:06

I am recieving a buffer which is having unicode and ansi character. How can I read it so that I can get the actual string.

Jochen Arndt6-Jul-12 0:22

6-Jul-12 0:22

Please be a little bit more specific (e.g. by providing an example). Do you have a buffer that may contain an ANSI or Unicode (Microsoft wide character UTF16-LE, UTF16-BE, or UTF-8) string and you want to detect the encoding? If so, do you know the length of the buffer?

john56326-Jul-12 0:39

6-Jul-12 0:39

Yes you are right and I know the lenght of buffer.

Jochen Arndt6-Jul-12 0:56

6-Jul-12 0:56

Wide char strings stored in files usually begin with a BOM (Byte Order Mark)[^].

If there is no BOM in your case, use the ANSI strlen() function and compare the result with the buffer length without NULL bytes. If it matches, the string is ANSI or UTF-8. If not, it is probably a wide char string. For Microsoft Unicode (UTF-16LE), this can be verified by comparing the return value of wcslen() with the buffer length (multiply returned length by two / sizeof(wchar_t)).

This solution requires that the string in the buffer is NULL terminated. If not, create a new buffer with length + 2, copy the data and append two NULL bytes (two bytes to have a valid terminated wide char string).

If the strings in the buffer are NULL terminated, you may also look for two NULL bytes at the end of UTF-16 strings.

john56326-Jul-12 1:09

6-Jul-12 1:09

Thanks for your help but I dont know how to check BOM..can you show by example...

Jochen Arndt6-Jul-12 1:18

6-Jul-12 1:18

Detecting code page from BOM:

C++

UINT nCP = 0; // CP_ACP
if (0 == memcmp(lpszBuf, "\xEF\xBB\xBF", 3))
    nCP = CP_UTF8;
else if (0 == memcmp(lpszBuf, "\xFF\xFE", 2))
    nCP = 1200; // UTF-16LE
else if (0 == memcmp(lpszBuf, "\xFE\xFF", 2))
    nCP = 1201; // UTF-16BE

However, this requires the presence of a BOM. With text files, there should be one for UTF-16 and may be one with UTF-8. If you get your data from elsewhere (e.g. as network package), there may be no BOM.

john56326-Jul-12 1:31

6-Jul-12 1:31

It will comapare the byte with 0xFFFE, 0xFEFF but my problem is to seperate out ANSI and UNICODE string from a BYTE buffer.

My buffer will will be like "ConnectInfo?ã„·ã„´??56.58.98.142?fgg?n?v HTT". In this buffer unicode data is "ã„·ã„´" and remaining data is in ANSI.
I need to seperate out the data and need to create a complete string of unicode characters.

Jochen Arndt6-Jul-12 1:52

6-Jul-12 1:52

That's why I asked for an example in my first post:
Your string seems to be UTF-8 encoded.

To decode it you can use MultiByteToWideChar() which converts it to a wide char string:

C++

// Assuming nLen is length including trailing NULL byte
// Get length of wide char string
int nWideLen = ::MultiByteToWideChar(CP_UTF8, 0, lpszBuf, nLen, NULL, 0);
wchar_t *lpszWide = new wchar_t[nWideLen];
::MultiByteToWideChar(CP_UTF8, 0, lpszBuf, nLen, lpszWide, nWideLen);
// Use string lpszWide
delete [] lpszWide;

BCN-1637-Jul-12 16:32

BCN-163

7-Jul-12 16:32

execuse me. where can I get some data about encoding. actualy, I don't know exactly what's the different between UTF8 and UNICODE. Thank you very much

Jochen Arndt7-Jul-12 22:28

7-Jul-12 22:28

Good starting points for such questions are Google and Wikipedia. For Unicode, they should both bring you to www.unicode.org.

Unicode is the standard and UTF-8 is one Unicode Transformation Format. Another encoding format is UTF-16LE which is used by Microsoft Windows (WCHAR and wchar_t types). When you read Unicode in Microsoft documentation, it refers mostly to the internally used UTF-16LE format.

BCN-1638-Jul-12 2:36

BCN-163

8-Jul-12 2:36

thank you very much. what I really want to know is the details which may have some help for work. I will visit the site, thanks again.

Programming Problem

Ahmed Charfeddine5-Jul-12 22:54

Ahmed Charfeddine

5-Jul-12 22:54

We have a collection of items C(x)
A thread A is responsible for performing tasks for those items in a continous and circular way.
In coding terms, we would write :

void doWork() // function executed by thread A
{
	while(true)
	{
		for each x in C(x)
			perform task(x).
	}
}

We have other threads, that write and remove items from C(x) :

void remove(x);
void Add(x);

Now comes the final specification :

The function task(x), returns either true or false. If false is returned, then it means, there no need to call task(x) for the same x
in the future, unless some thread makes a signal for us that the thread A should be processing all element x.

let's call this notification function :

void NotifyIncludeAllElements();

What is the best approach here ?

Push Framework - now released !
http://www.pushframework.com

Re: Programming Problem

«_Superman_»6-Jul-12 17:55

«_Superman_»

6-Jul-12 17:55

x can be a structure variable -

struct _tagX
{
    ... Xdata;
    ...
    bool process;
};

Now process can be used as a flag to decide if processing is needed.

«_Superman_»
I love work. It gives me something to do between weekends.

Microsoft MVP (Visual C++)

Polymorphism in C

Re: Programming Problem

Ahmed Charfeddine7-Jul-12 0:25

Ahmed Charfeddine

7-Jul-12 0:25

The size of collection C(X) can be so much big, that we might be interested in isolate the elements that are temporarily ignored, in another collection.

Further more, if all items are ignored, i am hoping that thread A blocks and not consume CPU for nothing.

Finally, the rate of calls to NotifyIncludeAllElements can be so much high, that if we make a shared lock with thread A there, then the performance of processing of A is greatly affected.

I tried to think about semaphore, signals, CAS operations, but came up with particular solution, you ca see it in this class I called AbstractDistributor : (Link to CodePlex)
http://pushframework.codeplex.com/SourceControl/changeset/view/98598#1909758[^]

When I think about the problem I can see many real world scenarios applicable to it. I am wondering if it is already solved in a good way.

Push Framework - now released !
http://www.pushframework.com

Format INT_MAX

_Flaviu5-Jul-12 22:53

5-Jul-12 22:53

I have follow code :

C++

m_sHex.Format(_T("FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF")); // max value for 32 bit

char* p = NULL;
const int nDec = strtol(m_sHex, &p, 16);
sUDec.Format(_T("%u"), nDec);
sDec.Format(_T("%d"), nDec);

and sUDec is '65535' and sDec is '65535' ... I expected to see sUDec 4.294.967.295 and sDec -1 ... what I'm missing ?

Jochen Arndt5-Jul-12 23:08

5-Jul-12 23:08

strtol() will stop parsing at the first character that is not a hex digit. In your string this is the first space character.

When you remove the spaces from your string, all will be parsed. However, this will produce an out of range error (errno is set to ERANGE and strtol() returns LONG_MAX). Because your string represents a 4 x 4 x 8 = 128 bit number.

To handle unsigned numbers, you may also use strtoul().

_Flaviu5-Jul-12 23:29

5-Jul-12 23:29

You are right ! Thank you !

David Crow6-Jul-12 2:38

David Crow

6-Jul-12 2:38

Flaviu2 wrote:
m_sHex.Format(_T("FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF")); // max value for 32 bit

Strange that you would use Format() rather than just assigning the value directly.

"One man's wage rise is another man's price increase." - Harold Wilson

"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons

"Show me a community that obeys the Ten Commandments and I'll show you a less crowded prison system." - Anonymous

_Flaviu6-Jul-12 4:09

6-Jul-12 4:09

I want to setup a default value if the string contain non haxa characters ...

Get Client Area Excluding Scroll Bar

David Crow6-Jul-12 4:41

David Crow

6-Jul-12 4:41

I realize that, but since you are not actually doing any formatting, why not just use:

m_sHex = _T("FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF");

AmbiguousName5-Jul-12 19:30

AmbiguousName

5-Jul-12 19:30

hello guys... I am trying to get the client area of my CListCtrl which shows vertical scroll bar when needed. The View is set to Report. Now, to prevent horizontal scrollbar, I am setting the width my column manually like this.

LVCOLUMN col;
CListCtrl->GetClientArea(&Rect1);
col.cx = (Rect1.right - Rect1.left) - 20;

But I want to set the width of my column based on something like this

CListCtrl->GetClientArea(&Rect1);
GetClientArea(ScrollBar of CListCtrl) // what to do to get client area of scroll bar?

LVCOLUMN col;
col.cx = (Rect1.right - Rect1.left) - Width of Vertical ScrollBar; // see?

But I don't have a clue how to get width of this built-in scrollbar OR how to get width of CListCtrl exluding ScrollBar? Thanks for any pointers.

This world is going to explode due to international politics, SOON.

Re: Get Client Area Excluding Scroll Bar

Jochen Arndt5-Jul-12 21:32

5-Jul-12 21:32

GetClientRect() (not GetClientArea) returns the client area of the list control. The area of optional scroll bars is not included in the returned rectangle. So GetClientRect() returns different rectangles when scroll bars are present or not.

The size of the vertical scroll bar can be determined using GetSystemMetrics(SM_CXVSCROLL). You may use this value upon list initialization (with empty list when no vertical scroll bar is shown) to preserve space for the scroll bar.

One instance of object

_Flaviu5-Jul-12 7:17