

Thank you , i was editing so the steps can be read properly....atleast i know that i was not wrong. i appricate you help and gives me confidence that peoople online can help me with my problems while i am learning.





also, i found that with this approach if i have number 99 the remainder will show as 1 and which would say it is a prime number, but again 99 is not a prime number.





What step is that at? Obviously you'd get a remainder of 1 for 2, but then in the next step you'd find that the remainder with 3 is 0, and therefore it's not a prime.





won't the remainder be 0 for 2, when you divided 2 by 2?





Sure, but you're not testing 2, you're testing 99, right? The remainder of 99 / 2 is 1, the remainder of 99 / 3 is 0





so assuming the number 2 in these steps in not properly defined and so if we move in the sequence and found another number to be divisible and remainder to be 0 (we should then in this case consider it to be correct) and therefore 99 would not show up as prime in when the algorithm runs, is that correct?





i found the soltion: The flag value can either be 1 or 0 in the course of this program. Flag is just a variable. This is tested in step 7 to determine if the number is prime or not prime. flag value is initially set to 1(during initialization). It may or may not change at step 6 depending on whether rem=0. The value of rem is 0 when the result of the calculation at step 5 is 0 (happens when the division produces no reminder). as you know by doing num mod i we divide num by the current value of i and check the reminder. If the reminder is 0 that means num is divisible by i. So num cannot be a prime. Whenever we find that reminder is 0 we set value of flag=0 which means the number is not prime.





the algorithm in plain english is
(0) initially set i=2 and flag=1
(1) take a number (num) which we want to test for being prime.
(2) Then we start dividing the number num successively by i= 2, 3, 4.....upto (num1) and check for the reminder. At each iteration we increase i by 1.
(3) At any stage, if we find that num is divisible by i then (num is not prime) set flag=0
(4) If we reach i= (num1) and still reminder is never 0 at any stage, then flag value remains unchanged at 1 and the number is prime
We test the flag value at the end of the program to decide whether num is prime or not.






I've prototyped a way to do pattern discovery using SQL, but I still have a poor understanding of where this method fits in the data mining vernacular. Being setbased, I'm not building a tree, although a functional tree does arise in the result set. The steps: 1) Build a "look up" table, by doing a cross join, yielding a combinatorial "dictionary" (a rainbow table) of ngram "words." 2) Get COUNT(*) > n , using SQL GROUP BY, matched against a large table of items 3) Further look for equivalent longer selfmatches within the result set. My seed table is 177 items, allowed to crossjoin itself 3x, for a final table 2.8 million 3gram words (takes about 35 seconds to build this table in Postgres). The actual itemset table is 10 million rows in series (serially numbered), although the actual number of itemsets might be considered smaller. I've recorded 35** seconds on the join between the two original tables, yielding all the simple repeating 3grams meeting group by's count(*) > x (that's the dictionary joined to the itemsets. That's the Q&D discovery step, and then subsequent steps simply apply a selfjoin for longerchained repeating series. These have been pretty quick, in the 50 millisecond range. My questions are: 1) What's the best way to describe this algorithm? Frequent pattern? Motif? 2) It's a simple enough method, but is it fast enough for general use? I.e. other data mining apps where performance requirements are different from my own? 3) I've wondered if SQL could be convinced to patternmatch like an LCS dynamic programming algorithm, by matching across gaps in the sequence, with maybe a lookup table of allowable variances & distance between matching values?
**Right now I'm seeing 50 seconds after the buffers load, but I reinstalled Postgres & my postgres.conf file apparently is all defaults now (the postgres process back to only 16 mb getting buffered, so it's suffering more I/O to the SSD drive). Thanks in advance,  Lee
 modified 5Sep14 0:17am.





Member 11060173 wrote: seed table is 177 items, allowed to crossjoin itself 3x, for a final table 2.8 million 3gram words
...
actual itemset table is 10 million rows Ehm, 2.8 million is about half of 177*177*177  that's quite a lot, but does still fit into memory (RAM). With 10 million rows, your trigram table will have some 10^20 rows, and that's beyond the memory of any machine nowadays, even beyond the capacity of any hard disk.
It won't work (already that factor of 10^14 applied to the present 35 seconds should tell you that).





Oh, I forgot to mention that the trigram dictionary is trimmed by abs(val1+val2+val3) <= 88 (it's a vector dataset of small int). But the 5.5m trigram dictionary might not slow things much given the use of covering indices (the access is all via btree indices, obviating the need for as much memory).
I looked into using a 4gram dictionary but it presented a very large table, much larger than the 3gram dictionary, and worse it made for more overlapping duplicates in building the equivalent of an FPtree (at least 1 extra overlap, whereas w/ the 3gram matches I'm always overlapping by n+2). Also I sense a trigrambased tree innately reflects the smallest useful vector of fromandto applicable.
One problem might be that in highfrequency datasets I could see an explosion of 3gram noise that doesn't always support better (longer) matches, bloating the output. I understand that in FPTree algo's there's a minimum support criterion that perhaps works around this. There may be a way to ameliorate this in SQL, such as checking for matching adjacencies in a manner that'll optimize via a correlated subquery, using ANY or [NOT] IN.
I haven't had enough time to fully experiment with various datasets, I've been going through a application language selection process** & am contemplating looking into PostGIS' geometric data & index features (RTree indices) as a way to get better, longer string matches, even perhaps supporting approximate or intermittent matches akin to the ability of LCS/cosine match algorithms (but on larger sets with more expressive syntax).
I'm prepared to start coding in C or Julia, but I'll avoid it if Postgres proves "fast enough." That's b/c as new data are imported to the DBMS I'll want to rerun pattern discoveries in the background against the main data store. My current 10 million rows are an exorbitant sample, outscaling anything I expect to encounter in the actual data (MIDI note vectors).
[Edit:]
Just found this:
https://www.academia.edu/5184801/SQL_Based_Frequent_Pattern_Mining_with_FPgrowth[^]
It's a very old paper (circa 2001?), but I'll probably be following their methodology. Maybe they gave up b/c DB/2 was too slow vs. algorithmic FPGrowth in C++.
Also, from a 2006 paper: http://webdocs.cs.ualberta.ca/~zaiane/postscript/adma05.pdf
"...In this work we presented COFIClosed, an algorithm for mining frequent
closed patterns. This novel algorithm is based on existing data structures FPtree
and COFItree. Our contribution is a new way to mine those existing structures
using a novel traversal approach. Using this algorithm, we mine extremely large
datasets, our performance studies showed that the COFIClosed was able to
mine efficiently 100 million transactions in less than 3500 seconds on a small
desktop while other known approaches failed..."
I know that's old hat by now (my 2008era Thinkpad T400 Celeron laptop w/ its 4GB RAM & SSD drive vs. his "small desktop"), but I'm in the ballpark.
**( As for fast application langs, esp. for other algorithmic jobs not best modeled in a SQL DBMS, I think I just found a winner: http://www.julialang.org[^] & http://juliawebstack.org/[^] )
modified 5Sep14 10:57am.





I've been playing around with binary search algorithms and have created a novel new variant that appears to be significantly faster than any traditional implementations I've seen.
int boundless_binary_search(int *array, int array_size, int key)
{
register int mid, i;
mid = i = array_size  1;
while (mid > 7)
{
mid = mid / 2;
if (key < array[i  mid]) i = mid;
}
while (i && key < array[i]) i;
if (key == array[i])
{
return i;
}
return 1;
}
I'm wondering if this is a novel binary search implementation, or has this approach been used by someone before?
Some variants and benchmark graphs:
https://sites.google.com/site/binarysearchcube/binarysearch[^]





Perhaps I'm not thinking about it right (I have a cold so I'm not particularly sharp right now), but it seems to me that if the item is in the second half of the array, it will essentially devolve into a linear search.





It switches to a linear search when roughly 8 elements are left. The i != 0 check is there in case the key is smaller than the value at index 0.





Yes ok, I guess the cold got to me. I was thinking something weird based around the assumption that mid started in the middle, which it obviously doesn't.
So it works, that's good. It seems closely related to the variant which keeps a "midpoint" and a "span" (here it's the next span (mid ), and the midpoint plus that next span (i )). Same pros and cons too (needs fixup at the end, but inner loop is simple), the "midpoint/span" variant is usually seen (when seen at all) in its "more complicated math in the inner loop"form which doesn't need fixup, but then what's the point.





Using a midpoint and a span is slower because it requires 2 assignments per loop opposed to 1.5 (on average) in my implementation.
I assume it's the fixup and assignment issue that left academics stumped for the past 60 years. There are also caching issues for larger arrays, for which I've created a variant that is mindful in that regard.





Gregorius van den Hoven wrote: requires 2 assignments per loop opposed to 1.5 (on average) in my implementation. Well you realize it'll be a conditional move, right? But only one, whereas regular binary search would have two (or an unpredictable branch, yuck).
So you'd get something like (from GCC output)
.L4:
sar edx
mov ecx, eax
sub ecx, edx
cmp [esi+ecx*4], ebx
cmovg eax, ecx
cmp edx, 7
jg .L4
In a quick test, GCC didn't feel like using cmov s for plain old "left and right bounds" binary search. You can easily measure a huge difference due to that sort of thing, and I'm not sure that's 100% fair, after all you could implement ye olde binary search with cmov s.





I'm not well versed in that area.
Looks like the compiler is doing something differently when you free up a registry space, but it's still slower.
int tailing_binary_search(int *array, int array_size, int key)
{
register int bot, mid, i;
if (key < array[0])
{
return 1;
}
bot = 0;
i = array_size  1;
mid = i / 2;
while (bot + 7 < i)
{
if (key < array[mid])
{
i = mid  1;
mid = (i  bot) / 2;
}
else
{
bot = mid;
mid += (i  bot) / 2;
}
}
while (key < array[i]) i;
if (key == array[i])
{
return i;
}
return 1;
}





Yea GCC at least makes some branches there in the loop, and it's using more complicated divisions by 2 (that work when negative).





I want to calculate the surface of an area on a sphere including projection effect.
For this I have a scanned image (which results in a 2D disc). On this image I can highlight an area and I have the index of each pixel of that area relative to the center of that disc (x, y coordinates). I also have the radius (in pixels and meters)
How can I calculate the correction needed for each pixel in order to take in account the projection effect (larger at the edges of the disc)?
Thanks.
[SOLUTION]
It's a pretty long solution, so I'll try to be as brief as possible.
1. You need to scale down the sphere to a unit sphere (radius=1)
2. With that you know that x²+y²+z² = 1 and thus z = sqrt(1x²y²) and the surface (of a full sphere) is 4*pi*r² = 4*pi, but you only have half the sphere so S=2*pi
3. What you need is the cosine of the angle of the pixel with the Zaxis. This is the z calculated in step 2 (the sqrt) divided by the radius which you reduced to one.
4. In the end you get this formula for the area per pixel: (1/r² * 1/(2*pi*sqrt(1x²y²)) ). If you loop through the pixels of the complete area you just need to to sum up all these pixels.
5. This results in a fraction of the surface compared to the sphere surface which you can then convert to any unit you like.
On first sight this "looks" correct
[/SOLUTION]
modified 8Aug14 7:50am.





I've been studying lock free algorithms, specifically the circular lock free ring buffer. I fully understand what they're doing, but am having trouble applying that to some of my code. It seems that the lock free ring buffer is meant for situations where the queue is neither starved nor saturated for any "significant" (in CPU terms) period of time.
For example, I recently had some code with multiple producers and a single consumer. When the server was running at a normal load, the lock free ring buffer would have been an excellent solution. However, when the server load was low it seems the consumer would just be spinning, using quite a bit of CPU doing nothing. Contrawise, when the server load was high, the producers would be spinning (in the current code, if this happens, the producers have a wait of a certain period at which point, they discard their data and grab the next set [losing data periodically was permissible since the next set would allow a full reconstruction, albeit with stutter]), but how long do they spin? Can't use just a count since time is important in this situation, but grabbing clock is expensive and simply looping the thread consumes a lot of CPU, depriving other producers of their quanta.
Am I missing something?





I posted a lockfree stack a few days ago. There would be no such issues of spinning/locking up the cpu. The only "problem" with it compared to the circular buffer you're using now is that the stack is FILO instead of FIFO.
A Fundamental LockFree Building Block  The LockFree Stack[^]
Hope it helps, lemme know if you have any questions.





Michael Gazonda wrote: There would be no such issues of spinning/locking up the cpu
This is the part I struggle with. If the stack is full, how does the CPU not spin? Likewise, if the stack is empty, pop returns false. Then what?





With a stack, there is no "full". It's like a singlelinked list. Each item points to the next, and carries with it the space required for doing so.
If the stack is empty, and returns false, then that's up to you to handle in whatever way is appropriate.
For myself, I wrote this to handle memory allocation (where I would manually allocate on false), or as a messaging system where false just meant there was no work to do and I would wait on a condition_variable.





