How take a random row from a PySpark DataFrame?

How can I get a random row from a PySpark DataFrame? I only see the method sample() which takes a fraction as parameter. Setting this fraction to 1/numberOfRows leads to random results, where sometimes I won't get any row.

On RRD there is a method takeSample() that takes as a parameter the number of elements you want the sample to contain. I understand that this might be slow, as you have to count each partition, but is there a way to get something like this on a DataFrame?

Answers


You can simply call takeSample on a RDD:

df = sqlContext.createDataFrame(
    [(1, "a"), (2, "b"), (3, "c"), (4, "d")], ("k", "v"))
df.rdd.takeSample(False, 1, seed=0)
## [Row(k=3, v='c')]

If you don't want to collect you can simply take a higher fraction and limit:

df.sample(False, 0.1, seed=0).limit(1)

Need Your Help

ASP.NET and Firefox: why doesn't clicking on a GridView ButtonField do anything?

asp.net javascript firefox

I have a pretty simple ASP.NET Web Form that looks a bit like the following:

How to tell if screen is on with ADB

android testing adb monkeyrunner

I am looking to find out whether or not it is possible to determine if the screen is on on an android device using ADB. I need to know this for some tests I am trying to run using monkey runner. Is...