XML / XSL

How to de-batch very large XMLDocument into bit size chunks

2-Mar-10 21:46

Yeah - I'd was just looking at it and realising that you needed some way of determining what indicated unique files...which you've done...

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

CodeProject MVP for 2010 - who'd'a thunk it!

ONeil Tomlinson28-Feb-10 7:39

Re: How to de-batch very large XMLDocument into bit size chunks

28-Feb-10 7:39

I have a very large XMLDocument with 56,000 child records each having 8 child field elements. The application that i have written performs a complicated process on the XML. It can process 10,000 records in 5 minutes and 20,000 in 25 minutes (increases exponentially) but takes maybe 10 hours to process the 56,000 record (which is unacceptable). Whats the best way to de-batch this XML Document into bite size chunks (eg 10,000 and after 5 iteration will process 50,000. which will execute in 25-30 minutes processing time)

Thanks.

daveyerwin28-Feb-10 8:44

daveyerwin

28-Feb-10 8:44

How many megabytes is your xml file?
Does the "complicated process" operate on
only one of the 56,000 children independantly,
without refferencing the other children ?
Show me a childNode and tell me about
this "complicated process".
How much free memory does your computer have available ?

Re: How to de-batch very large XMLDocument into bit size chunks

ONeil Tomlinson1-Mar-10 6:33

1-Mar-10 6:33

There is a sample of the child record. And there are 56,000 of them in the xml document. I can realy give you the full details of what the complicated process is doing but part of it is using the Muenchian Method grouping on 'cost_centre' and 'expense_code'.
The app is runningon a dev box 2.50GH, 2GB.
It has 1/2GB free when app is not running

<Transaction> 
<Entry>
    <payroll_id>payroll_id_0</payroll_id>
    <payroll_nm>payroll_nm_0</payroll_nm>
    <payRunDate>payRunDate_0</payRunDate>
    <payRunID>payRunID_0</payRunID>
    <empno>10</empno>
    <surname>surname_0</surname>
    <forename>forename_0</forename>
    <pos_ref>pos_ref_0</pos_ref>
    <pos_name>pos_name_0</pos_name>
    <element_id>element_id_0</element_id>
    <element_name>element_name_0</element_name>
    <cost_centre>c11</cost_centre>
    <expense_code>e11</expense_code>
    <debit_credit>debit_credit_0</debit_credit>
    <amount>10</amount>
    <signed_amount>10</signed_amount>
  </Entry>
<Entry>
....
 </Entry>
</Transaction>

Unclosed tr tag

John Jones II26-Feb-10 4:35

John Jones II

26-Feb-10 4:35

Why is this invalid?

<xsl:choose>
<xsl:when test="position() mod 2 = 1">
    <tr bgcolor="Silver">
</xsl:when>
<xsl:otherwise>
    <tr>
</xsl:otherwise>
</xsl:choose>
...
</tr>

Internet Explorer reports the error "End tag 'xsl:when' does not match the start tag 'tr'"

Re: Unclosed tr tag

Richard MacCutchan26-Feb-10 5:23

Richard MacCutchan

26-Feb-10 5:23

The line:

<tr bgcolor="Silver">

is unclosed, so the next closing tag should be </tr> rather than </xsl:when>.

txtspeak is the realm of 9 year old children, not developers. Christian Graus

Re: Unclosed tr tag

Stuart Dootson26-Feb-10 7:45

XML schema - Element name used to identify item problem... - need to use the element name to identify value when importing into SQL 2005 tables

26-Feb-10 7:45

The previous answer told you why the error occurred - here's a solution:

<tr>
<xsl:if test="position() mod 2 = 1">
    <xsl:attribute name="bgcolor">Silver</xsl:attribute>
</xsl:if>
...
</tr>

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

CodeProject MVP for 2010 - who'd'a thunk it!

VicarH25-Feb-10 7:37

VicarH

25-Feb-10 7:37

I am working on a project for a State Agency where the data is received from multiple private entities and loaded into SQL 2005 tables. The XML schema was designed and published before my involvement with the project and now it cannot be changed. One of the elements to receive (Charges) occurs multiple times within a client but each is for a different concept. Instead of using an attribute to identify the reason for the Charge, they used the element name to do it. For example:

<FERTILIZER_CHARGE>2300</FERTILIZER_CHARGE> -- 20 different charge types identified

instead of

<CHARGE TYPE=”FERTILIZER” AMOUNT=”2300” />

I need to move all the charges to a single table and identify each one with its type but cannot find any examples where the element name is used to update a column in a table. PLEASE HELP!!!

Display a image in picturebox

kpuneeth723-Feb-10 20:36

23-Feb-10 20:36

i just want to load one image to the picturebox, who's path is stored in xml file. please tell me how to do this.
GIve examples.

Stuart Dootson23-Feb-10 20:50

Re: Display a image in picturebox [modified]

23-Feb-10 20:50

kpuneeth7 wrote:
GIve examples

Without knowing a little about the structure of your XML file and what technology you're using, that would be difficult to do...

But it's simply a case of reading the XML file with an XML parser (think XmlReader in .NET), and finding the node containing the image file's path.

The rest of it (displaying the image in the picture box) is nothing to do with XML.

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

CodeProject MVP for 2010 - who'd'a thunk it!

kpuneeth723-Feb-10 21:01

23-Feb-10 21:01

I am using a simple xml file.....

<path>C:\Documents and Settings\Imran\Desktop\Picture\apple.jpg

And i am using visual studio 2008
C# language.
.net framework

modified on Wednesday, February 24, 2010 3:20 AM

Stuart Dootson23-Feb-10 21:36

23-Feb-10 21:36

Here's a short C# program that will read the bit of XML you want and write it to the console.

using System.Xml.XPath;
using System.IO;

namespace ConsoleApplication1
{
   class Program
   {
      static void Main(string[] args)
      {
         // The XML file is specified as the first command line argument
         // imagePath will contain the image path contained in the XML file
         string imagePath = new XPathDocument(new StreamReader(args[0])).CreateNavigator().SelectSingleNode("/image/img").Value;
         System.Console.WriteLine(imagePath);
      }
   }
}

I've used XPath to navigate to the required element in the XML file (that's the .SelectSingleNode("/image/img") bit) and read the text there (the .Value bit). Simples.

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

CodeProject MVP for 2010 - who'd'a thunk it!

kpuneeth723-Feb-10 23:53

23-Feb-10 23:53

Thanks for the code....

BUt its giving indexoutofrange exception......

HElp

Stuart Dootson24-Feb-10 0:08

24-Feb-10 0:08

You need to give a command line argument that specifies the XML file...the code comments DO say that....

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

CodeProject MVP for 2010 - who'd'a thunk it!

kpuneeth724-Feb-10 1:26

Multiple group using Muenchian Method [modified]

24-Feb-10 1:26

Thanks dude..... Smile | :)

I got it

Thank u very much

ONeil Tomlinson23-Feb-10 4:15

Re: Multiple group using Muenchian Method

23-Feb-10 4:15

I have the following input XML. I want to group all unique Code/ID combination.

<List>
 <Section>
  	<Code>AA</Code>
  	<ID>11</ID>
 <Section>
 <Section>
  	<Code>AA</Code>
  	<ID>11</ID>
 <Section>
 <Section>
  	<Code>CC</Code>
  	<ID>11</ID>
 <Section>
 <Section>
  	<Code>AA</Code>
  	<ID>22</ID>
 <Section>
 <Section>
  	<Code>AA</Code>
  	<ID>11</ID>
 <Section>
<List>

So result should be as shown below.

<Code>AA</Code>
<ID>11</ID>

<Code>CC</Code>
<ID>11</ID>

<Code>AA</Code>
<ID>22</ID>

Im using the Muenchian Method. Currely my XSL (below) is grouping by Code (and not Code/ID combination)

<?xml version="1.0" encoding="UTF-16"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:var="http://schemas.microsoft.com/BizTalk/2003/var" exclude-result-prefixes="msxsl var" version="1.0" xmlns:ns0="http://A4C.Interface.HRP.Payroll.Schemas">
	<xsl:output omit-xml-declaration="yes" method="xml" version="1.0" />

	<xsl:key name="CCids" match="/List/Section" use="Code"/>

	<xsl:template match="/">
		<xsl:apply-templates select="/ns0:Payroll" />
	</xsl:template>
	<xsl:template match="/ns0:Payroll">
		<ns0:List>
			<xsl:for-each select="/List/Section [generate-id(.)=generate-id(key('CCids',Code))]">
				<Section>
					<Code>
						<xsl:value-of select="code/text()" />
					</Code>
					<ID>
						<xsl:value-of select="id/text()" />
					</ID>
				</Section>
			</xsl:for-each>
		</ns0:List>
	</xsl:template>
</xsl:stylesheet>

How do i get this to use a Code/ID combination? I tried nested for-each loop but no luck. Thanks

modified on Tuesday, February 23, 2010 11:40 AM

Stuart Dootson23-Feb-10 20:47

Re: Multiple group using Muenchian Method

23-Feb-10 20:47

Firstly, your key needs to have an expression which reflects what you want to retrieve:

<xsl:key name="CCids" match="/List/Section" use="concat(Code,'|',ID)"/>

And then you need to use that same expression when accessing items referenced by the key.

Oh - and in addition, I would suggest not using a for loop - it's not idiomatic XSL. Here's the file I'd use:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:var="http://schemas.microsoft.com/BizTalk/2003/var" exclude-result-prefixes="msxsl var" version="1.0" xmlns:ns0="http://A4C.Interface.HRP.Payroll.Schemas">
   <xsl:output omit-xml-declaration="yes" method="xml" version="1.0" encoding="utf-8"/>

   <xsl:key name="CCids" match="/List/Section" use="concat(Code,'|',ID)"/>

   <xsl:template match="/">
      <ns0:List>
         <xsl:apply-templates select="/List/Section [generate-id(.)=generate-id(key('CCids',concat(Code,'|',ID)))]"/>
      </ns0:List>
   </xsl:template>
            
   <xsl:template match="Section">
      <Section>
         <Code>
            <xsl:value-of select="Code/text()" />
         </Code>
         <ID>
            <xsl:value-of select="ID/text()" />
         </ID>
      </Section>
   </xsl:template>
</xsl:stylesheet>

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

CodeProject MVP for 2010 - who'd'a thunk it!

ONeil Tomlinson24-Feb-10 0:05

How Do I deserialize the following Xml string? [modified]

24-Feb-10 0:05

Thanks..Really helpful Wink | ;)

ashishtango22-Feb-10 23:05

ashishtango

22-Feb-10 23:05

Hi,
I am getting this error There is an error in XML document (1, 2)
When i Deserialize following Xml.

<EnvironmentConfigurations>
<EnvironmentConfiguration>
<EnvironmentName>ABC</EnvironmentName>
<CMSConfiguration>
<Address>net.tcp://localhost:9000/Manager</Address>
<Binding>netTcpBinding</Binding>
<Contract>Common.Interfaces.Server</Contract>
</CMSConfiguration>
</EnvironmentConfiguration>
<EnvironmentConfiguration>
<EnvironmentName>XYZ</EnvironmentName>
<CMSConfiguration>
<Address>net.tcp://localhost:9000/Manager</Address>
<Binding>netTcpBinding</Binding>
<Contract>Common.Interfaces.Server</Contract>
</CMSConfiguration>
</EnvironmentConfiguration>
</EnvironmentConfigurations>

modified on Tuesday, February 23, 2010 5:11 AM

Stuart Dootson22-Feb-10 23:54

22-Feb-10 23:54

Deserialize to what? The way in which it's deserialized depends on the type you think it is.

Are you sure this was serialized from .NET? That's not the only way to read and write XML, you know - maybe you should be using an XMLReader and parsing the XML yourself?

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

CodeProject MVP for 2010 - who'd'a thunk it!

ashishtango23-Feb-10 16:52

ashishtango

23-Feb-10 16:52

I have serialized using XmlSerializer.
I have removed Xml version and namespace information from the XML.
I have changed the root node from
<ArrayOfEnvironmentConfigurations> to <EnvironmentConfigurations>
I want to desrialize this xml.
Thank you.

Stuart Dootson23-Feb-10 20:27

23-Feb-10 20:27

The answer to your problem is very, very clear - don't mess with the XML that the serializer produces - all of the things you changed are in there for a reason!!!!

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

CodeProject MVP for 2010 - who'd'a thunk it!