Comment by tuxdna on Transform a large dataframe - takes too long
@Amoxz here you go - gist.github.com/tuxdna/d6b16610e65e6c4ab05c70057518dbe8
View ArticleComment by tuxdna on Scala can't access Java inner class?
This incompatibility is bugging me now. Has anyone found an alternative approach other than reflection?
View ArticleComment by tuxdna on Execute command on all files in a directory
@HubertKario You may want to read more about -print0 for find and -0 for xargs which use null character instead of any whitespace ( including newlines ).
View ArticleComment by tuxdna on Merge two tables in Scala/Spark
@skan: Try this - stackoverflow.com/questions/8820778/linux-join-2-csv-files
View ArticleComment by tuxdna on Binary Search in large .txt with python (ordered by hash)
The first part ( before colon ) is clearly 20 byte sha1 hash, but what is :27 in your sample?
View ArticleComment by tuxdna on Binary Search in large .txt with python (ordered by hash)
your file must be accessible by the user running web-server
View ArticleComment by tuxdna on How can I scrape faster
Did you first try to benchmark how long it takes to process single json? Assuming it takes 300ms per record, you can process all of these records sequentially in about 5 days.
View ArticleComment by tuxdna on How do I calculate fuzz ratio between two columns?
First, how are you calculating fuzz ratio between two strings?
View ArticleComment by tuxdna on I don't understand why I get a "too many values to...
Updated with EDIT1 for your reference.
View ArticleComment by tuxdna on How to measure GPU memory usage of TensorFlow model
What is the memory consumption now?
View ArticleComment by tuxdna on How to resample a pandas data frame every half second...
This solution is neat!
View ArticleComment by tuxdna on How to efficient copy an Array to another in Scala?
Even though b is val which prevents us from reassigning it to some other value, Array is mutable.
View ArticleClik here to view.
Number of threads in Akka keep increasing. What could be wrong?
Why does the thread count keep increasing ?LOOK AT THE BOTTOM RIGHT in this image.The overall flow is like this:Akka HTTP Server API -> on http request, sendMessageTo DataProcessingActor ->...
View ArticleAnswer by tuxdna for Number of threads in Akka keep increasing. What could be...
I figured out that ElasticSearch was the problem. I am using Java API for ElasticSearch, and that is leaking sockets because of the way it was being used from Java API. Now resolved as described...
View ArticleAnswer by tuxdna for reduce key, list(values) to key, value using scala
In the absence of concrete types in the question I have made an assumption that the values in your array are Char and Int tuples respectively. Here is how we can transform to the desired...
View ArticleAnswer by tuxdna for Primitive types to AnyRef in Scala
That is because, AnyRef is for objects and AnyVal is for primitives. You can use an Array[Any] in your case:var s: PreparedStatement = session.prepare("insert into test_person (name, age, point) values...
View ArticleAggregate Pandas DataFrame based on condition that uses multiple columns?
import pandas as pddata = {"K": ["A", "A", "B", "B", "B"],"LABEL": ["X123", "X123", "X21", "L31", "L31"],"VALUE": [1, 3, 1, 2, 5.0]}df = pd.DataFrame.from_dict(data)output = """ K LABEL VALUE0 A X12...
View ArticlePrettify JSON data using Ruby on the terminal
I have earlier used Python for doing pretty output of JSON data like this:python -mjson.tool input.jsonI wanted to get similar output using Ruby. I am doing it like this:ruby -rrubygems -e 'require...
View ArticleString range in Scala
In Ruby we can do this:$ irb>> ("aa".."bb").map { |x| x }=> ["aa", "ab", "ac", "ad", "ae", "af", "ag", "ah", "ai", "aj", "ak", "al", "am", "an", "ao", "ap", "aq", "ar", "as", "at", "au", "av",...
View ArticleR DataFrame - One Hot Encoding of column containing multiple terms [duplicate]
I have a dataframe with a column having multiple values ( comma separated ):mydf <- structure(list(Age = c(99L, 10L, 40L, 15L), Info = c("good, bad, sad", "nice, happy, joy", "NULL", "okay, nice,...
View Article
