What do you mean by overkill?
This HDFS has its limitation and does not locate the processing logic power?
I have no personal experience to this product
As per summarized:
Hadoop is an Apache Software Foundation distributed file system and data management project with goals for storing and managing large amounts of data. Hadoop uses a storage system called HDFS to connect commodity personal computers, known as nodes, contained within clusters over which data blocks are distributed. You can access and store the data blocks as one seamless file system using the MapReduce processing model.
HDFS shares many common features with other distributed file systems while supporting some important differences. One significant difference is HDFS's write-once-read-many model that relaxes concurrency control requirements, simplifies data coherency, and enables high-throughput access.
In order to provide an optimized data-access model, HDFS is designed to locate processing logic near the data rather than locating data near the application space.
What do you mean by overkill...Hadoop is an Apache Software
Foundation distributed file system and data management project with goals for
storing and managing large amounts of data.
Your stated requirements do not meet the definition of "large amounts of data".
Let me give you some examples of large data
- 2000 transactions a second sustained with a expected lifetime of 7 years and a real time need of 6 to 18 months immediate availability. Each transaction has a 1k size.
- Each originator will produce several 100 meg downloads several times a month. Sizing must expect up to 10,000 originators with a lifetime of 5 years.
It just hit me when reading this post that your problem is similar to to the problem of splitting a large file into chunks. As an example if you have big archieve on disk and want to store it on removable devices e.g. floppy, cdrom or dvd.
In the old days we used ARJ to split a compressed file into a number of volumes (.arj, .a01, .a02, etc.). Each volume had a fixed maximum size e.g. 1.44 MB for a floppy.
A limitation was that you could not add/remove stuff from .a02 without breaking the big file, so if you need to do this, splitting big files into compressed volumes might not be your solution.
I'm fairly new to writing unit tests and I've run into something of a design/architecture question.
I'm operating under the belied that any given method should not be overly large, and any that is should be refactored into smaller methods.
As a result I end up with this:
public MyClass : IMyClass
// some code
// some other code
Obviously this is grossly simplified, but it serves my purposes for this question. I have my classes loosly couples so when writing a unit test I can stub any interface that is injected into MyClass and isolate the code under test.
How do I go about stubbing Method1 and Method2? I'm using Moles (and can't change because of company restrictions), but I suspect/hope this is a testing platform independent question.
Should I be desiging this different? Not using private methods, but public virtual would allow me more flexibility, but doesn't feel like the right approach.
Any advice or pointers would be greatly appreciated.
Last Visit: 11-Dec-18 4:18 Last Update: 11-Dec-18 4:18