Category Archives: Python

Article online!

Yesterday, a new publication for which I programmed the GIS analysis and co-authored, has been put online:

Heuner, M., Weber, A., Schröder, U., Kleinschmit, B., Schröder, B. (2016). Facilitating political decisions using species distribution models to assess restoration measures in heavily modified estuaries. Marine Pollution Bulletin 110(1), 250–260. DOI: 10.1016/j.marpolbul.2016.06.056.

With the supplementary material of data, R- and Python-scripts you can reproduce the analysis.

DSL Geschwindigkeiten

Da ich in den vergangenen Wochen einen neuen DSL-Vertrag abgeschlossen habe, bei dem vermehrt Probleme bei der Erreichbarkeit meines privaten Webservers auftraten, habe ich beschlossen Verbindungsgeschwindigkeiten und Erreichbarkeit zu überwachen. Basierend auf diesem Blogpost und dem zugrundeliegenden Python-Tool zur Messung von Verbindungsgeschwindigkeiten von Janis Jansons habe ich eine dauerhafte Geschwindigkeitsüberwachung, die alle 10 Minute ausgeführt wird, eingerichtet.
Da die produzierte *.csv-Datei allerdings reichlich unübersichtlich ist, habe ich ein kleines Skript in R geschrieben, um die Daten, die in die *.csv-Datei geschrieben werden, zu visualisieren. Zunächst muss also R installiert werden. Unter Ubuntu erfolgt das mit folgendem Befehl:
sudo apt-get install r-base
Danach speichert man das folgende Skript lokal unter ~/scripts/speedtest_plot.R ab.

### set locale
Sys.setlocale("LC_TIME", "en_US.utf8")

### set working directory
setwd("/home/arnd/")

### import data
data.df <- read.table("speedtest.csv", header=FALSE, sep=",", dec=".", 
                      stringsAsFactors=FALSE, fill=TRUE, na.strings=c("[]", ""))

### preprocess the data
# name the columns
colnames(data.df) <- c("time", "download", "upload", "unit", "server")
# strptime
data.df$time <- strptime(data.df$time, format="%a, %d %b %Y %H:%M:%S %z", tz="")
# down- & upload
data.df$download[which(data.df$download == -1)] <- NA
data.df$upload[which(data.df$upload == -1)] <- NA
# manipulate server names => remove brackets
data.df$server <- gsub("[", "", data.df$server, fixed=TRUE)
data.df$server <- gsub("]", "", data.df$server, fixed=TRUE)
data.df$server <- gsub("'", "", data.df$server, fixed=TRUE)
#unique(data.df$server)
# create and factorize date
data.df$date <- strptime(data.df$time, format="%Y-%m-%d")
data.df$fdate <- as.factor(as.character(data.df$date))

### plot
pdf("speedtest_plot.pdf")
par(font.lab=2)
plot(data.df$time, data.df$download, type="l", col="blue", lwd=2, xlab="Zeit", 
     ylab="Mbit", main="DSL-Geschwindigkeiten", ylim=c(0,100), xaxt="n")
r <- as.POSIXct(round(range(data.df$time), "days"))
axis.POSIXct(1, at=seq(r[1], r[2], by="day"), format="%d.%m.%y", labels=TRUE)
lines(data.df$time, data.df$upload, col="red", lwd=2)
points(data.df$time[which(is.na(data.df$download) == TRUE)], 
       rep(100, length(which(is.na(data.df$download) == TRUE))), pch=19, 
       cex=0.1, col="blue")
text(data.df$time[ceiling(length(data.df$time)/2)], 97.4, 
     "Zeitpunkte ohne Messung", cex=0.7)
points(data.df$time[which(is.na(data.df$upload) == TRUE)], 
       rep(95, length(which(is.na(data.df$upload) == TRUE))), pch=19, cex=0.1, 
       col="red")
text(data.df$time[1], 55, "DOWNLOAD", col="blue", cex=1.5, pos=4)
text(data.df$time[1], 5, "UPLOAD", col="red", cex=1.5, pos=4)
abline(h=50, lty=3, lwd=0.5)
mtext(paste0("Aktualisiert: ", strftime(Sys.time(), "%d.%m.%Y %H:%M:%S")), 
      cex=0.7)
dev.off()

png("speedtest_plot.png")
par(font.lab=2)
plot(data.df$time, data.df$download, type="l", col="blue", lwd=2, xlab="Zeit", 
     ylab="Mbit", main="DSL-Geschwindigkeiten", ylim=c(0,100), xaxt="n")
r <- as.POSIXct(round(range(data.df$time), "days"))
axis.POSIXct(1, at=seq(r[1], r[2], by="day"), format="%d.%m.%y", labels=TRUE)
lines(data.df$time, data.df$upload, col="red", lwd=2)
points(data.df$time[which(is.na(data.df$download) == TRUE)], 
       rep(100, length(which(is.na(data.df$download) == TRUE))), pch=19, 
       cex=0.1, col="blue")
text(data.df$time[ceiling(length(data.df$time)/2)], 97.4, 
     "Zeitpunkte ohne Messung", cex=0.7)
points(data.df$time[which(is.na(data.df$upload) == TRUE)], 
       rep(95, length(which(is.na(data.df$upload) == TRUE))), pch=19, cex=0.1, 
       col="red")
text(data.df$time[1], 55, "DOWNLOAD", col="blue", cex=1.5, pos=4)
text(data.df$time[1], 5, "UPLOAD", col="red", cex=1.5, pos=4)
abline(h=50, lty=3, lwd=0.5)
mtext(paste0("Aktualisiert: ", strftime(Sys.time(), "%d.%m.%Y %H:%M:%S")), 
      cex=0.7)
dev.off()

###
# subset to print the data only for the last month
# select the last month:
# 60 sec * 60 min * 24 hours * 30 days
# 2592000 = one month
data.df <- subset(data.df, time > max(data.df$time) - 2592000)

png("speedtest_plot_last_month.png", height=720, width=720)
par(font.lab=2)
plot(data.df$time, data.df$download, type="l", col="blue", lwd=2, xlab="Zeit", 
     ylab="Mbit", main="DSL-Geschwindigkeiten", ylim=c(0,100), xaxt="n")
r <- as.POSIXct(round(range(data.df$time), "days"))
axis.POSIXct(1, at=seq(r[1], r[2], by="day"), format="%d.%m.%y", labels=TRUE)
lines(data.df$time, data.df$upload, col="red", lwd=2)
points(data.df$time[which(is.na(data.df$download) == TRUE)], 
       rep(100, length(which(is.na(data.df$download) == TRUE))), pch=19, 
       cex=0.1, col="blue")
text(data.df$time[ceiling(length(data.df$time)/2)], 97.4, 
     "Zeitpunkte ohne Messung", cex=0.8)
points(data.df$time[which(is.na(data.df$upload) == TRUE)], 
       rep(95, length(which(is.na(data.df$upload) == TRUE))), pch=19, cex=0.1, 
       col="red")
text(data.df$time[1], 55, "DOWNLOAD", col="blue", cex=1.5, pos=4)
text(data.df$time[1], 5, "UPLOAD", col="red", cex=1.5, pos=4)
abline(h=50, lty=3, lwd=0.5)
mtext(paste0("Aktualisiert: ", strftime(Sys.time(), "%d.%m.%Y %H:%M:%S")), 
      cex=0.8)
dev.off()

###
# subset to print the data only for the last week
# select the last week:
# 60 sec * 60 min * 24 hours * 7 days
# 640800 = one week
data.df <- subset(data.df, time > max(data.df$time) - 604800)

png("speedtest_plot_last_week.png", height=720, width=720)
par(font.lab=2)
plot(data.df$time, data.df$download, type="l", col="blue", lwd=2, xlab="Zeit", 
     ylab="Mbit", main="DSL-Geschwindigkeiten", ylim=c(0,100), xaxt="n")
r <- as.POSIXct(round(range(data.df$time), "days"))
axis.POSIXct(1, at=seq(r[1], r[2], by="day"), format="%d.%m.%y", labels=TRUE)
lines(data.df$time, data.df$upload, col="red", lwd=2)
points(data.df$time[which(is.na(data.df$download) == TRUE)], 
       rep(100, length(which(is.na(data.df$download) == TRUE))), pch=19, 
       cex=0.1, col="blue")
text(data.df$time[ceiling(length(data.df$time)/2)], 97.4, 
     "Zeitpunkte ohne Messung", cex=0.8)
points(data.df$time[which(is.na(data.df$upload) == TRUE)], 
       rep(95, length(which(is.na(data.df$upload) == TRUE))), pch=19, cex=0.1, 
       col="red")
text(data.df$time[1], 55, "DOWNLOAD", col="blue", cex=1.5, pos=4)
text(data.df$time[1], 5, "UPLOAD", col="red", cex=1.5, pos=4)
abline(h=50, lty=3, lwd=0.5)
mtext(paste0("Aktualisiert: ", strftime(Sys.time(), "%d.%m.%Y %H:%M:%S")), 
      cex=0.8)
dev.off()
q("no")

Und führt es anschliessend ebenso zeitgesteuert, wie den Speedtest, als Cronjob aus. Dazu öffnet man mit crontab -e die Crontab und fügt folgende Zeile ein:
1 * * * * Rscript ~/scripts/speedtest_plot.R 2>&1 /dev/null
Dieser Befehl, der stündlich ausgeführt wird, produziert drei unterschiedliche Plots, von denen einer – nämlich der mit den Ergebnissen der Geschwindigkeitsmessungen der letzten Woche – hier dargestellt ist. Hinter dem verlinkten Bild findeet sich eine interaktive R Shiny-Animation:

Geschwindigkeitsdaten der letzten Woche

file system mapping

To map a file system and export the product into a spreadsheet I wrote the following short and handy Python script using the glob-module. The mapped drive and mapping depth can be determined by editing line 12. Line 16 determines the location, where the produced *.csv-File will be saved, and line 17 determines the column-separator of the *.csv-file.

This script was developed as part of my employment at the Federal Institute of Hydrology (BfG) in Koblenz, Germany.

#!/opt/Python-2.7.6/bin/python
# -*- coding: utf-8 -*-
# author: arnd.weber(at)bafg.de
# date: 2014.09.01

###
# load python modules
import os.path, glob

###
# glob.glob object with a depth of 4 levels, starting with the drive letter C: 
Depth4 = glob.glob(r"C:/*/*/*")

###
# export the list of folders (Depth4) to a file
filepath = r"C:/Users/arnd/Desktop/folders.csv"
separator = r";"
with open(filepath, "w") as file:
    for an_entry in Depth4:
        if os.path.isdir(an_entry):
            file.write(an_entry.replace(r"/",separator).replace(r"\\",separator) + r"\n")

###
# end the script and exit
exit(0)