Email Password Remember me?  Lost your password?

Introduction

A long time ago, in a very remote island known as Lilliput, society was split into two factions: Big-Endians who opened their soft-boiled eggs at the larger end ("the primitive way") and Little-Endians who broke their eggs at the smaller end. As the Emperor commanded all his subjects to break the smaller end, this resulted in a civil war with dramatic consequences: 11.000 people have, at several times, suffered death rather than submitting to breaking their eggs at the smaller end. Eventually, the 'Little-Endian' vs. 'Big-Endian' feud carried over into the world of computing as well, where it refers to the order in which bytes in multi-byte numbers should be stored, most-significant first (Big-Endian) or least-significant first (Little-Endian) to be more precise [1]

• Big-Endian means that the most significant byte of any multibyte data field is stored at the lowest memory address, which is also the address of the larger field.
• Little-Endian means that the least significant byte of any multibyte data field is stored at the lowest memory address, which is also the address of the larger field.

For example, consider the 32-bit number, 0xDEADBEEF. Following the Big-Endian convention, a computer will store it as follows:

Figure 1. Big-Endian: The most significant byte is stored at the lowest byte address.

Whereas architectures that follow the Little-Endian rules will store it as depicted in Figure 2:

Figure 2. Little-endian: Least significant byte is stored at the lowest byte address.

The Intel x86 family and Digital Equipment Corporation architectures (PDP-11, VAX, Alpha) are representatives of Little-Endian, while the Sun SPARC, IBM 360/370, and Motorola 68000 and 88000 architectures are Big-Endians. Still, other architectures such as PowerPC, MIPS, and Intel�s 64 IA-64 are Bi-Endian, i.e. they are capable of operating in either Big-Endian or Little-Endian mode. [1].

Endianess is also referred to as the NUXI problem. Imagine the word UNIX stored in two 2-byte words. In a Big-Endian system, it would be stored as UNIX. In a little-endian system, it would be stored as NUXI.

Which format is better?

Like the egg debate described in the Gulliver's Travels, the Big- .vs. Little-Endian computer dispute has much more to do with political issues than with technological merits. In practice, both systems perform equally well in most applications. There is however a significant difference in performance when using Little-Endian processors instead of Big-Endian ones in network devices (more details below).

How to switch from one format to the other?

It is very easy to reverse a multi-byte number if you need the other format, it is simply a matter of swapping bytes and the conversion is the same in both directions. The following example shows how an Endian conversion function could be implemented using simple C unions:

unsigned long ByteSwap1 (unsigned long nLongNumber)
{
   union u {unsigned long vi; unsigned char c[sizeof(unsigned long)];}; 
   union v {unsigned long ni; unsigned char d[sizeof(unsigned long)];};
   union u un; 
   union v vn; 
   un.vi = nLongNumber; 
   vn.d[0]=un.c[3]; 
   vn.d[1]=un.c[2]; 
   vn.d[2]=un.c[1]; 
   vn.d[3]=un.c[0]; 
   return (vn.ni); 
}

Note that this function is intented to work with 32-bit integers.

A more efficient function can be implemented using bitwise operations as shown below:

unsigned long ByteSwap2 (unsigned long nLongNumber)
{
   return (((nLongNumber&0x000000FF)<<24)+((nLongNumber&0x0000FF00)<<8)+
   ((nLongNumber&0x00FF0000)>>8)+((nLongNumber&0xFF000000)>>24));
}

And this is a version in assembly language:

unsigned long ByteSwap3 (unsigned long nLongNumber)
{
   unsigned long nRetNumber ;

   __asm
   {
      mov eax, nLongNumber
      xchg ah, al
      ror eax, 16
      xchg ah, al
      mov nRetNumber, eax
   }

   return nRetNumber;
}

A 16-bit version of a byte swap function is really straightforward:

unsigned short ByteSwap4 (unsigned short nValue)
{
   return (((nValue>> 8)) | (nValue << 8));

}

Finally, we can write a more general function that can deal with any atomic data type (e.g. int, float, double, etc) with automatic size detection:

#include <algorithm> //required for std::swap


#define ByteSwap5(x) ByteSwap((unsigned char *) &x,sizeof(x))

void ByteSwap(unsigned char * b, int n)
{
   register int i = 0;
   register int j = n-1;
   while (i<j)
   {
      std::swap(b[i], b[j]);
      i++, j--;
   }
}

For example, the next code snippet shows how to convert a data array of doubles from one format (e.g. Big-Endian) to the other (e.g. Little-Endian):

double* dArray; //array in big-endian format

int n; //Number of elements


for (register int i = 0; i <n; i++) 
   ByteSwap5(dArray[i]);

Actually, in most cases, you won't need to implement any of the above functions since there are a set of socket functions (see Table I), declared in winsock2.h, which are defined for TCP/IP, so all machines that support TCP/IP networking have them available. They store the data in 'network byte order' which is standard and endianness independent.

Table I: Windows Sockets Byte-Order Conversion Functions [2]

The socket interface specifies a standard byte ordering called network-byte order, which happens to be Big-Endian. Consequently, all network communication should be Big-Endian, irrespective of the client or server architecture.

Suppose your machine uses Little Endian order. To transmit the 32-bit value 0x0a0b0c0d over a TCP/IP connection, you have to call htonl() and transmit the result:

TransmitNum(htonl(0x0a0b0c0d)); 

Likewise, to convert an incoming 32-bit value, use ntohl():

int n = ntohl(GetNumberFromNetwork()); 

If the processor on which the TCP/IP stack is to be run is itself also Big-Endian, each of the four macros (i.e. ntohs, ntohl, htons, htonl) will be defined to do nothing and there will be no run-time performance impact. If, however, the processor is Little-Endian, the macros will reorder the bytes appropriately. These macros are routinely called when building and parsing network packets and when socket connections are created. Serious run-time performance penalties occur when using TCP/IP on a Little-Endian processor. For that reason, it may be unwise to select a Little-Endian processor for use in a device, such as a router or gateway, with an abundance of network functionality. (Excerpt from reference [1]).

One additional problem with the host-to-network APIs is that they are unable to manipulate 64-bit data elements. However, you can write your own ntohll() and htonll() corresponding functions:

• ntohll: converts a 64-bit integer to host byte order.
• ntonll: converts a 64-bit integer to network byte order.

The implementation is simple enough:

#define ntohll(x) (((_int64)(ntohl((int)((x << 32) >> 32))) << 32) | 
                     (unsigned int)ntohl(((int)(x >> 32)))) //By Runner

#define htonll(x) ntohll(x)

How to dynamically test for the Endian type at run time?

As explained in Computer Animation FAQ, you can use the following function to see if your code is running on a Little- or Big-Endian system:

#define BIG_ENDIAN      0
#define LITTLE_ENDIAN   1

int TestByteOrder()
{
   short int word = 0x0001;
   char *byte = (char *) &word;
   return(byte[0] ? LITTLE_ENDIAN : BIG_ENDIAN);
}

This code assigns the value 0001h to a 16-bit integer. A char pointer is then assigned to point at the first (least-significant) byte of the integer value. If the first byte of the integer is 0x01h, then the system is Little-Endian (the 0x01h is in the lowest, or least-significant, address). If it is 0x00h then the system is Big-Endian.

Similarly,

bool IsBigEndian()
{
   short word = 0x4321;
   if((*(char *)& word) != 0x21 )
     return true;
   else 
     return false;
}

which is just the reverse of the same coin.

You can also use the standard byte order API�s to determine the byte-order of a system at run-time. For example:

bool IsBigEndian() { return( htonl(1)==1 ); }

Auto detecting the correct Endian format of a data file

Suppose you are developing a Windows application that imports Nuclear Magnetic Resonance (NMR) spectra. High resolution NMR files are generally recorded in Silicon or Sun Workstations but recently Windows or Linux based spectrometers are emerging as practical substitutes. It turns out that you will need to know in advance the Endian format of the file to parse correctly all the information. Here are some practical guidelines you can follow to decipher the correct Endianness of a data file:

1. Typically, the binary file includes a header with the information about the Endian format.
2. If the header is not present , you can guess the Endian format if you know the native format of the computer from which the file comes from. For instance, if the file was created in a Sun Workstation, the Endian format will most likely be Big-Endian. 
3. If none of the above points apply, the Endian format can be determined by trial and error. For example, if after reading the file assuming one format, the spectrum does not make sense, you will know that you have to use the other format.

If the data points in the file are in floating point format (double), then the _isnan() function can be of some help to determine the Endian format. For example:

double dValue;
FILE* fp;
(...)
fread(&nValue, 1, sizeof(double), fp);
bool bByteSwap = _isnan(dValue) ? true : false

Note that this method does only guarantee that the byte swap operation is required if _isnan() returns a nonzero value (TRUE); if it returns 0 (FALSE), then it is not possible to know the correct Endian format without further information.

Acknowledgments

Thanks to Santiago Dom�nguez, Ehsan Akhgari, Santiago Fraga and Ignacio Sordo for their helpful suggestions.

References

1. Introducction to Endianness, by Michael Barr, Embedded Systems Programming.
2. Visual C++ Concepts: Adding Functionality. Windows Sockets: Byte Ordering

Further reading

• ON HOLY WARS AND A PLEA FOR PEACE by Danny Cohen.
• For a full technical discussion of software Endianness issues, refer to the Sun white paper, "Endianness in Solaris".
• A list of graphics common file formats and their Endian order can be found here.

You must Sign In to use this message board.

Per page 102550

FirstPrevNext

Question Prafulla Tekawade 8:06 11 Jan '07   Assume that an integer pointer x is declared in a "C" program as int *x; Further assume that the location of the pointer is 1000 and it points to an address 2000 where value 500 is stored in 4 bytes. What is the output of printf("%d",*x); a) 500 b) 1000 c) undefined d) 2000 What will be the answer? I think it should be ( c) bcoz on m/c will little endian sys it will print 500 & those with big endian it will print 0 I am not pretty sure. Assume x retrives two bytes of memory Please help. Sign In·View Thread·PermaLink Detecting big/little endian at compile time? Patrick Hoffmann 12:58 8 Sep '06   Does anybody know if there is a possibility to detetct big/little endian within the preprocessor? (Best Regards,) Patrick Hoffmann Sign In·View Thread·PermaLink 1.25/5 errors in macro Anonymous 13:10 3 Oct '05   #define ntohll(x) (((_int64)(ntohl((int)((x << 32) >> 32))) << 32) | (unsigned int)ntohl(((int)(x >> 32)))) //By Runner Don't forget the backslash to splice the preprocessor lines together: #define ntohll(x) (((_int64)(ntohl((int)((x << 32) >> 32))) << 32) | \ (unsigned int)ntohl(((int)(x >> 32)))) Whenever an argument is used in the body of a macro, write parentheses around it: #define ntohll(x) (((_int64)(ntohl((int)(((x) << 32) >> 32))) << 32) | \ (unsigned int)ntohl(((int)((x) >> 32)))) In a sensible world, ntohll ought to deal with an yield unsigned integral types. Never programmed on Windows, so I don't know about whether it is sensible in this way: #define ntohll(x) (((_uint64)(ntohl((unsigned int)(((x) << 32) >> 32))) << 32) | \ (unsigned int)ntohl(((unsigned int)((x) >> 32)))) For portability, the right types to used are those provided by : #define ntohll(x) (((uint64_t)(ntohl((uint32_t)(((x) << 32) >> 32))) << 32) | \ (uint32_t)ntohl(((uint32_t)((x) >> 32)))) It is not really safe to trust the type of an argument. Cast it: #define ntohll(x) (((uint64_t)(ntohl((uint32_t)(((uint64_t)(x) << 32) >> 32))) << 32) | \ (uint32_t)ntohl(((uint32_t)((uint64_t)(x) >> 32)))) The first chunk shifts the low-order half up, then down, then up. Makes me dizzy: #define ntohll(x) (((uint64_t)(ntohl((uint32_t)(((uint64_t)(x))))) << 32) | \ (uint32_t)ntohl(((uint32_t)((uint64_t)(x) >> 32)))) I'm confused by the parens: 123 33 45 5567 77 76543 2 #define ntohll(x) (((uint64_t)(ntohl((uint32_t)(((uint64_t)(x))))) << 32) | \ (uint32_t)ntohl(((uint32_t)((uint64_t)(x) >> 32)))) 2 2 234 445 55 5 4321 #define ntohll(x) ((uint64_t)ntohl((uint32_t)(uint64_t)(x)) << 32 | \ (uint32_t)ntohl((uint32_t)((uint64_t)(x) >> 32))) A couple of casts are pointless: #define ntohll(x) ((uint64_t)ntohl((uint32_t)(x)) << 32 | \ ntohl((uint32_t)((uint64_t)(x) >> 32))) There might be more problems, even in my transformations. But it looks better to me. Sign In·View Thread·PermaLink 5.00/5 What about left-to-right or right-to-left.? rbid 22:16 8 Jan '05   Hello, Great article.. From my experience, some people confuse two terms: - Endianes vs. Memory view. Memory can be described as "left-to-right" or "right-to-left". If we have the value of 0x01020304 stored as a 32bit value, according to Endianes and memory view we can get: Big-Endian 01 02 03 04 ... left-to-right A A+1 A+2 A+3 A+4 Little-Endian 01 02 03 04 ... right-to-left A+3 A+2 A+1 A A+4 Little-Endian 04 03 02 01 ... left-to-right A A+1 A+2 A+3 A+4 'A' represents a byte-address on the memory. In another words, when you describe a 32bit register, some people put the MSB on the right and the LSB on the left or vice-versa, for describing a number with the same byte ordering (Endianes) Just wanted to help. Have a nice day. Chau! -- Ricky Marek (AKA: rbid) -- "Things are only impossible until they are not" --- Jean-Luc Picard Sign In·View Thread·PermaLink incorrect double value reading from binary file anonymous 23:08 8 Nov '04   hello i m reading wrong double value from binary file through vc++ but correct value from vb. why it happens..? purushottam. Sign In·View Thread·PermaLink incorrect reading double value .... Anonymous 23:00 8 Nov '04   hello, i m reading binary file which contains double data which is little-endian but it gives wrong value through vc++, but gives correct values from vb, why it happens...? purushottam. Sign In·View Thread·PermaLink SIMD Optimized SwapEndian function for large Unicode Strings immo 10:28 26 Aug '03   This function is designed to swap endian of large memory buffer (reprezented by "memory" class in this listing) extremaly fast, using various code for different CPUs (you must use CPUID function or sth like it, look at "cpu_info" class). static void cvSwapEndian() { // copyright by Piotr Sawicki // please send me info if You are using this code: immo@medialab.pl if ( !memory.GetSize() ) return; #ifdef CV_ASSEMBLY_OFF if ( cpu_info.dwFeatures & CV_CPU_MMX ) { int len = (memory.GetSize() >> 3) + 1; __m64 *src = (__m64*)(BYTE*)memory; src += len; len = -len; do src[len] = _m_por(_m_psllwi(src[len], 8), _m_psrlwi(src[len], 8)); while ( ++len != 0 ); _m_empty(); } else { int len = (memory.GetSize() >> 2) + 1; DWORD *src = (DWORD*)(BYTE*)memory; src += len; len = -len; do src[len] = ((src[len] >> 8) & 0x00ff00ff) | ((src[len] << 8) & 0xff00ff00); while ( ++len != 0 ); } #else // CV_ASSEMBLY_OFF if ( cpu_info.dwFeatures & CV_CPU_SSE2 ) { DWORD src = (DWORD)(PBYTE)memory; DWORD len = (memory.GetSize() + 64) & -64; __asm { mov esi, DWORD PTR [src] mov ecx, DWORD PTR [len] lea esi, DWORD PTR [esi+ecx] neg ecx align 16 $sse2_simd: mov eax, DWORD PTR [esi+ecx+64] ; prefetching movdqa xmm0, XMMWORD PTR [esi+ecx+0*16] ; mov double quad movdqa xmm2, XMMWORD PTR [esi+ecx+1*16]; movdqa xmm4, XMMWORD PTR [esi+ecx+2*16]; movdqa xmm6, XMMWORD PTR [esi+ecx+3*16]; movdqa xmm1, xmm0 movdqa xmm3, xmm2 movdqa xmm5, xmm4 movdqa xmm7, xmm6 psllw xmm1, 8 ; packed shift left logical word psrlw xmm0, 8 ; packed shift right logical word por xmm1, xmm0 ; packed or psllw xmm3, 8 psrlw xmm2, 8 por xmm3, xmm2 psllw xmm5, 8 psrlw xmm4, 8 por xmm5, xmm4 psllw xmm7, 8 psrlw xmm6, 8 por xmm7, xmm6 movdqa XMMWORD PTR [esi+ecx+0*16], xmm1 movdqa XMMWORD PTR [esi+ecx+1*16], xmm3 movdqa XMMWORD PTR [esi+ecx+2*16], xmm5 movdqa XMMWORD PTR [esi+ecx+3*16], xmm7 add ecx, 64 jnz $sse2_simd } } else if ( cpu_info.dwFeatures & CV_CPU_MMX ) { DWORD src = (DWORD)(PBYTE)memory; DWORD len = (memory.GetSize() + 32) & -32; __asm { mov esi, DWORD PTR[src] mov ecx, DWORD PTR[len] shr ecx, 3 lea esi, DWORD PTR[esi+ecx*8] neg ecx align 16 $mmx_simd: mov eax, DWORD PTR[esi+ecx*8+32] ; prefetching movq mm0, MMWORD PTR[esi+ecx*8] ; mov quad movq mm2, MMWORD PTR[esi+ecx*8+8]; movq mm4, MMWORD PTR[esi+ecx*8+16]; movq mm6, MMWORD PTR[esi+ecx*8+24]; movq mm1, mm0 movq mm3, mm2 movq mm5, mm4 movq mm7, mm6 psllw mm1, 8 ; packed shift left logical word psrlw mm0, 8 ; packed shift right logical word por mm1, mm0 ; packed or psllw mm3, 8 psrlw mm2, 8 por mm3, mm2 psllw mm5, 8 psrlw mm4, 8 por mm5, mm4 psllw mm7, 8 psrlw mm6, 8 por mm7, mm6 movq MMWORD PTR[esi+ecx*8], mm1 movq MMWORD PTR[esi+ecx*8+8], mm3 movq MMWORD PTR[esi+ecx*8+16], mm5 movq MMWORD PTR[esi+ecx*8+24], mm7 add ecx, 4 jnz $mmx_simd emms; ; empty mmx flag } } else { DWORD src = (DWORD)(PBYTE)memory; DWORD len = (memory.GetSize() + 4) & -4; __asm { mov esi, DWORD PTR[src] mov ecx, DWORD PTR[len] shr ecx, 2 lea esi, DWORD PTR[esi+ecx*4] neg ecx align 16 $386_simd: mov eax, DWORD PTR[esi+ecx*4] mov edx, eax shr eax, 8 shl edx, 8 xor eax, edx and eax, 00ff00ffH xor eax, edx mov DWORD PTR[esi+ecx*4], eax inc ecx jnz $386_simd } } #endif // CV_ASSEMBLY_OFF } Piotr Sawicki home page: http://www.medialab.pl/cvservice/ GCS/IT d-(+) s+: a-- C++ UL++ !P L++@ !E W+++$ N+ o+ !K w+++ !O M- !V PS+ PE+ Y+ !PGP t 5? X+ R tv- b++ DI+ D++ G e++>+++ h--- r+++ y+++* Sign In·View Thread·PermaLink An error in macro ntohll(x) Runner 7:59 20 Aug '03   I think it should not be : #define ntohll(x) ((x << 32) | (unsigned int)ntohl(((int)(x >> 32)))) You need swap the other 32bits word of the double too, so the correct one is : #define ntohll(x) (((_int64)(ntohl((int)((x << 32) >> 32))) << 32) | (unsigned int)ntohl(((int)(x >> 32)))) Sign In·View Thread·PermaLink Re: An error in macro ntohll(x) Juan Carlos Cobas 9:21 20 Aug '03   Yes, you're right. I'll update the article Sign In·View Thread·PermaLink Re: An error in macro ntohll(x) Anonymous 7:28 23 Mar '05   This won't work on a big endian system as it will swap the upper and lower bits producing a 3rd number that is off. Using the revised Macro #define ntohll(x) (((_int64)(ntohl((int)((x << 32) >> 32))) << 32) | (unsigned int)ntohl(((int)(x >> 32)))) Little Endian System 01234567 | 89ABCDEF <- Start 89ABCDEF | 01234567 <- Order Swap FEDCBA98 | 76543210 <- ntohl Big Endian System 01234567 | 89ABCDEF <- Start 89ABCDEF | 01234567 <- Order Swap 89ABCDEF | 01234567 <- ntohl In reality the macro should do nothing on a big endian system, so be forwarned. the below macro should be fine for handling bigendian and little endian system as long as ntohl is functioning properly. It is more complex, so you may just wish to actually create a bitswap function if you intend to use this extensivly. #define ntohll(x) (ntohl((int)(x >> 32)) == (int)(x >> 32)) ? x : (((_int64)(ntohl((int)((x << 32) >> 32))) << 32) | (unsigned int)ntohl(((int)(x >> 32)))) Sign In·View Thread·PermaLink 2.00/5 Re: An error in macro ntohll(x) dotanb 5:09 17 Sep '08   There is a bug in this test: if x is a number which is small than 4G, than the test is wrong. I would have check if: (1) == ntohl(1) to prevent any dependency of the value of x. Dotan Barak Sign In·View Thread·PermaLink Re: An error in macro ntohll(x) Anonymous 9:49 21 Jun '05   Performance can also be approximately doubled by using addition rather than "or" (which has to check all the bits). #define ntohll(x) (((_int64)(ntohl((int)((x << 32) >> 32))) << 32) + (unsigned int)ntohl(((int)(x >> 32)))) Sign In·View Thread·PermaLink Faster assembly JohnAtTopcon 4:03 20 Aug '03   inline __int16 FlipBytes(__int16 nData) { __asm mov ax, [nData]; __asm bswap eax; __asm shr eax, 16; __asm mov [nData], ax; // you can take this out but leave it just to be sure return(nData); } inline __int32 FlipBytes(__int32 nData) { __asm mov eax, [nData]; __asm bswap eax; __asm mov [nData], eax; // you can take this out but leave it just to be sure return(nData); } Sign In·View Thread·PermaLink Re: Faster assembly Juan Carlos Cobas 5:29 20 Aug '03   Yep, they're surely faster under Intel platforms (cause the bswap instruction) Thanks for your comment Carlos Sign In·View Thread·PermaLink Re: Faster assembly Anonymous 19:50 9 May '05   There isn't a 16-bit version of the bswap instruction purely because they reccomend you do it this way instead: inline __int16 FlipBytes(__int16 nData) { __asm mov ax, [nData]; __asm xchg al, ah; __asm mov [nData], ax; // you can take this out but leave it just to be sure return(nData); } Sign In·View Thread·PermaLink Use of Union in ByteSwap1 ReorX 1:35 20 Aug '03   ByteSwap1 may return undefined values, depending on the compiler. It is not allowed to use two or more aspects of a union within one function. The Saviour of the World is a Penguin and Linus Torvalds is his Prophet. Sign In·View Thread·PermaLink 1.00/5 Nice ;) Kochise 20:56 19 Aug '03   http://www.codeproject.com/cpp/bitbashing.asp?df=100&forumid=3830&exp=0&app=50&select=266839#xx266839xx Kochise In Cod we trust ! Sign In·View Thread·PermaLink Re: Nice ;) Juan Carlos Cobas 23:19 19 Aug '03   Kochise wrote: http://www.codeproject.com/cpp/bitbashing.asp?df=100&forumid=3830&exp=0&app=50&select=266839#xx266839xx Good text Sign In·View Thread·PermaLink 2.00/5

Last Visit: 10:11 14 Mar '10     Last Update: 10:11 14 Mar '10 1