Deadlines are valuable, and deadlines are expensive.

Organized systems and societies need deadlines. It would be impossible to efficiently build a house if the subcontractors could deliver their goods or services whenever it were convenient for them. Movie studios and book publishers schedule their releases months in advance to allow distribution teams to plan their work. Software is dependent on subsystems that have to be in place before the entire program can work.

Along with the value that synchronized deliverables create, there are also real costs. Not simply the organizational cost of a missed deadline, but the significant damage to a reputation or brand that happens when a promise isn’t kept. And there’s a human cost–the stress and strain that comes from working to keep a promise that we might not have personally made, or that might be more difficult because someone else didn’t perform their part of the dance.

In the wide-open race for attention and commitments, the standards of deadlines have been wavering. For forty years, Saturday Night Live has gone on at 11:30. Not, as its creator says, because it’s ready, but because it’s 11:30. That’s the deal.

On Kickstarter, this sort of sacrosanct deadline is rare indeed. “This charger will ship in six weeks!” they say, when actually, it’s been more than a year with no shipment date in sight. Or with venture capitalists and other backers. “We’re going to beat the competition to market by three months.” Sometimes it feels like if the company doesn’t bring wishful thinking to the table, they won’t get funded. Given that choice, it’s no wonder that people get desperate. Wishful thinking might not be called lying, but it is. We should know better.

Earning the reputation as someone (a freelancer, a marketer, a company, a leader) who doesn’t miss a deadline is valuable. And it doesn’t happen simply because you avoid sleeping and work like a dog. That’s the last resort of someone who isn’t good at planning.

Here are some basic principles that might help with the planning part:

1. If you’re competing in an industry where the only way to ‘win’ is to lie about deadlines, realize that competing in that industry is a choice, and accept that you’re going to miss deadlines and have to deal with the emotional overhead that comes with that.
2. Knowing that it’s a choice, consider picking a different industry, one where keeping deadlines is expected and where you can gain satisfaction in creating value for others by keeping your promises.
3. Don’t rely on false deadlines as a form of incentive. It won’t work the same on everyone, which means that some people will take you at your word and actually deliver on time, while others will assume that it was simply a guideline. It’s more efficient to be clear and to help people understand from the outset what you mean by a deadline. The boy cried wolf but the villagers didn’t come.
4. At the same time, don’t use internal deadlines as a guaranteed component of your external promises. A project with no buffers is certain to be late. Not just likely to be late, but certain. Better buffers make better deadlines.
5. Embrace the fact that delivering something on a certain date costs more than delivering it whenever it’s ready. As a result, you should charge more, perhaps a great deal more, for the value that your promise of a deadline creates. And then spend that money to make sure the deadline isn’t missed.
6. Deadlines aren’t kept by people ‘doing their best.’ Keeping a deadline requires a systemic approach to dependencies and buffers and scenario planning. If you’re regularly cutting corners or burning out to meet deadlines, you have a systems problem.
7. The antidote to feature creep isn’t occasional pruning. That’s emotionally draining and a losing battle. The answer is to actively restructure the spec, removing or adding entire blocks of work. “That will be in the next version,” is a totally acceptable answer, particularly when people are depending on this version to ship on time.
8. A single deadline is a deadline that will certainly not be met. But if you can break down your big deadline into ten or fifteen intermediate milestones, you will know about your progress long before it’s too late to do something about it.
9. The Mythical Person-Month is a serious trap. Nine people, working together in perfect harmony, cannot figure out how to have a baby in one month. Throwing more people at a project often does not speed it up. By the time you start to solve a deadline problem this way, it might be too late. The alternative is to staff each component of your project with the right number of people, and to have as many components running in parallel as possible.
10. Bottlenecks are useful, until they aren’t. If you need just one person to approve every element of your project, it’s unlikely you can run as many things in parallel as you could. The alternative is to have a rigorous spec created in advance, in which many standards are approved before you even begin the work.
11. Discussions about timing often devolve into issues of trust, shame and effort. That’s not nearly as helpful as separating conversations about system structure and data from the ones about commitment and oomph.
12. Hidden problems don’t get better. In a hyper-connected world, there’s no technical reason why the project manager can’t know what the team in the field knows about the state of the project.

Like most things that matter, keeping deadlines is a skill, and since it’s a skill, we can learn it.

[More on this in my next post on what to do if you can’t avoid breaking your promise.]

As a data scientist, or as a leader of a data science team, you know the power and flexibility that open source data science delivers. However, if your team works within a typical enterprise, you compete for budget and executive mindshare with a wide variety of other analytic tools, including self-service BI and point-and-click data science tools. Navigating this landscape, and convincing others in your organization of the value of open source data science, can be difficult. In this blog post, we draw on our recent webinar on this topic to give you some talking points to use with your colleagues when tackling this challenge.

However, it is important to keep in mind that “code-first” does not mean “code only.” While code is often the right choice, most organizations need multiple tools, to ensure you have the right tool for the task at hand.

What makes programming languages like R and Python great for making data applications is the wealth of contributed extension packages that supercharge app development. You can turn your code into an interactive web app with not much extra code once you have a workflow and an interesting question.

We have reviewed Docker basics and how to dockerize a very simple Shiny app. For anything that is a little bit more complex, you will have to manage dependencies. Dependency management is one of the most important aspects of app development with Docker. In this post you will learn about different options.

Workflow

In our world today, COVID-19 data needs no introduction. There are countless dashboards out there showing case counts in space and time. This app is no different. You can find all the R code associated with this post in this GitHub repository:

analythium/covidapp-shiny

A simple Shiny app to display and forecast COVID-19 daily cases – analythium/covidapp-shiny

GitHubanalythium

Download or clone the repository and open the 01-workflow directory. Now install/load some packages (forecast, jsonlite, ggplot2, and plotly), source the functions.R file. The workflow looks like this:

pred %
    get_data() %>%
    process_data(
    	cases = "confirmed", 
        last = "2021-05-01") %>%
    fit_model() %>%
    predict_model(
    	window = 30, 
        level = 95)

1. pick a country (the available slugified country codes are explained in the source file),
2. get the data from a daily updated web interface (JSON API),
3. process the raw data: what kinds of cases (confirmed/deaths) to consider and what should be the last day of the time series,
4. fit time series model to the data,
5. forecast x days following the last day of the time series and show prediction intervals.

The data source is the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University. The flat files provided by the CSSE are further processed to provide a JSON API (read more about the API and its endpoints, or explore the data interactively here).

We use exponential smoothing (ETS) as a time series forecasting method from the forecast package. There are many other time series forecasting methods (like ARIMA etc.). We picked ETS because of its ease of use for our demonstration purposes.

We can visualize the pred object as plot_all(pred) which returns a ggplot2 object like this one:

Turn the ggplot2 object into an interactive plotly graph as ggplotly(plot_all(pred)).

Shiny app

Change to the 02-shiny-app folder which has the following files:

.
├── README.md
├── app
│   ├── functions.R
│   ├── global.R
│   ├── server.R
│   └── ui.R
└── covidapp.Rproj

Run the app locally as shiny::runApp("app"). It will look like this with controls for country, case type, time window, prediction interval, and a checkbox to switch between the ggplot2 or plotly output types:

Play around with the app then let's move on to putting it in a container.

Explicit dependencies in Dockerfile

The first approach is to use RUN statements in the Dockerfile to install the required packages. Check the Dockerfile in the 03-docker-basic folder. The structure of the Dockerfile follows the general pattern outlined in this post. We use the rocker/r-ubuntu:20.04 base image and specify the RStudio Package Manager (RSPM) CRAN repository in Rprofile.site so that we can install binary packages for speedy Docker builds. Here are the relevant lines:

FROM rocker/r-ubuntu:20.04
...
COPY Rprofile.site /etc/R
...
RUN install.r shiny forecast jsonlite ggplot2 htmltools
RUN Rscript -e "install.packages('plotly')"
...

Required packages are installed with the littler utility install.r (littler is installed on all Rocker base images). You can also use Rscript to call install.packages(). There are other options too, like install2.r from littler, or using R -q -e install.packages() – -q suppresses the startup message, -e executes an expression then quits.

Build and test the image locally, use any image name you like (in export IMAGE=""), then visit http://localhost:8080 to see the app:

# name of the image
export IMAGE="analythium/covidapp-shiny:basic"

# build image
docker build -t $IMAGE .

# run and test locally
docker run -p 8080:3838 $IMAGE

Use DESCRIPTION file

The second approach is to record the dependencies in the DESCRIPTION file. You can find the example in the 04-docker-deps folder. The DESCRIPTION file contains basic information about an R package. The file states package dependencies and is used when installing the packages and its dependencies. The install_deps() function from the remotes package can install dependencies stated in a DESCRIPTION file. The DESCRIPTION file used here is quite rudimentary but it states the dependencies to be installed nonetheless:

Imports:
  shiny,
  forecast,
  jsonlite,
  ggplot2,
  htmltools,
  plotly

Use the same Ubuntu based R base image and the RSPM CRAN repository. Install the remotes package, copy the DESCRIPTION file into the image. Call remotes::install_deps() which will find the DESCRIPTION file in the current directory. Here are the relevant lines from the Dockerfile:

FROM rocker/r-ubuntu:20.04
...
COPY Rprofile.site /etc/R
...
RUN install.r remotes
COPY DESCRIPTION .
RUN Rscript -e "remotes::install_deps()"
...

Build and test the image as before, but use a different tag:

# name of the image
export IMAGE="analythium/covidapp-shiny:deps"

# build image
docker build -t $IMAGE .

# run and test locally
docker run -p 8080:3838 $IMAGE

Use the renv R package

The renv package is a versatile dependency management toolkit for R. You can discover dependencies with renv::init() and occasionally save the state of these libraries to a lockfile with renv::snapshot(). The nice thing about this approach is that the exact version of each package is recorded that makes Docker builds reproducible.

Switch to the 05-docker-renv directory and inspect the Dockerfile. Here are the most important lines (Focal Fossa is the code name for Ubuntu Linux version 20.04 LTS that matches our base image):

FROM rocker/r-ubuntu:20.04
...
RUN install.r remotes renv
...
COPY ./renv.lock .
RUN Rscript -e "options(renv.consent = TRUE); \
	renv::restore(lockfile = '/home/app/renv.lock', repos = \
    c(CRAN='https://packagemanager.rstudio.com/all/__linux__/focal/latest'))"
...

We need the remotes and renv packages. Then copy the renv.lock file, call renv::restore() by specifying the lockfile and the RSPM CRAN repository. The renv.consent = TRUE option is needed because this is a fresh setup (i.e. not copying the whole renv project).

Tag the Docker image with :renv and build:

# name of the image
export IMAGE="analythium/covidapp-shiny:renv"

# build image
docker build -t $IMAGE .

# run and test locally
docker run -p 8080:3838 $IMAGE

Comparison

We built the same Shiny app in three different ways. The sizes of the three images differ quite a bit, with the :renv image being 40% bigger that the other two images:

$ docker images --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}'

REPOSITORY                  TAG                 SIZE
analythium/covidapp-shiny   renv                1.7GB
analythium/covidapp-shiny   deps                1.18GB
analythium/covidapp-shiny   basic               1.24GB

The :basic image has 105 packages installed (try docker run analythium/covidapp-shiny:basic R -q -e 'nrow(installed.packages())'). The  :deps image has remotes added on top of these, the :renv image has remotes, renv and BH as extras. BH seems to be responsible for the size difference, this package provides Boost C++ header files. The COVID-19 app works perfectly fine without BH. In this particular case, this is a price to pay for the convenience of automatic dependency discovery provided by renv.

The renv package has a few different snapshot modes. The default is called “implicit”. This mode adds the intersection of all your installed packages and those used in your project as inferred by renv::dependencies() to the lockfile. Another mode, called “explicit”, which only capture packages which are listed in the project DESCRIPTION file. For the COVID-19 app, both these resulted in identical lockfiles. You can use renv::remove("BH") to remove BH from the project or use the “custom” model and list all the packages to be added to the lockfile.

If you go with the other two approaches, explicitly stating dependencies in the Dockerfile or in the DESCRIPTION file, you might end up missing some packages at first. These approaches might needs a few iterations before getting the package list just right.

Another important difference between these approaches is that renv pins the exact package versions in the lockfile. If you want to install versioned packages, use the remotes::install_version() function in the Dockerfile. The version-tagged Rocker images will by default use the MRAN snapshot mirror associated with the most recent date for which that image was current.

Summary

You learnt the basics of dependency management for Shiny apps with Docker. Now you can pick and refine an approach that you like most (there is no need to build the same app multiple ways).

Of course there is a lot more to talk about from different base images to managing system dependencies for the R packages. We'll cover that in an upcoming post.

Further reading

• Using renv with Docker
• Pin package versions by Roman Luštrik
• An Introduction to Rocker describing versioned images and more
• The Rockerverse: a recent update on R+Docker

SharePoint R integration and analysis, this tutorial is a continuation of the previous post Data analysis in R pdftools & pdftk. In the last post, we discussed one of the common data storages is pdf. Today we are going to discuss SharePoint R integration and analysis.

Nowadays most companies are depending on a common centralized database, such kind of situation SharePoint is one of the important tools for storing information.

Why Microsoft SharePoint?

It’s very handy and easy to use, can handle a huge amount of data, easy retrieval, and fast processing. One centralized database will remove all kinds of personal dependencies and no need to worry about data security.

Imagine if the employee resigned from the company all the data saved in one centralized platform and the newcomer can easily pick it up and contribute to it immediately.

Looking for Data Science jobs?

SharePoint R integration and analysis

In this tutorial, we are going to discuss following important steps.

Step 1: How login into the SharePoint database in r?

Step 2:- How to extract the data from SharePoint?

Step 3:- How to clean the data in r?

Step 4:- Analyze the data in r

Step 5:- Make a report and mail it to the respective person

We are not going to concentrate much on Step 4 and Step 5 because it’s subjective and varies based on the requirements.

How to login into the SharePoint database in r?

First, we need to save the user id and password in the mentioned format.

Usepassword



Once you store the user id and password, the next step is to set up the URL.



First go to the list you want to extract the information and take the URL from the browser, for example, if you want to extract the information from the “HR” list, the URL should be something like this.



http://your.sharepoint.websitename//HR//_vti_bin//owssvr.dll?XMLDATA=1&



The next step is to extract the list code, view code, and row limit from the share point database and join to the above URL.



The URL finally looks like mentioned format.



url



List code, View code and Row Limit looks something like this



LIST={alphanumericcodes}&VIEW={%alphanumericcodes}&RowLimit=something”



Let see how to find the list code? first, you need to click on the list setting and select “audience target setting“, from the browser URL now you can extract the list codes.



Differences between Association & Correlation



In the same way, you can extract view codes also from the list setting. Go to list settings and under view click “all items”, now from the URL you can extract view codes.



Finally, the row limit format looks something like this



RowLimit=&RootFolder=%2fmodulename%2fLists%2flistname



For extracting module name and list name, you can just click on the list link (the information you want to extract from) and from the browser, can extract the details.



Your URL ready now.



url



Getting Data



library(xml)
library(xlsx)
data



Now entire columns information’s saved in mydata



The column names should look like ows$Title or ows$category etc… now you can do the proper column renaming according to you and select relevant columns.



The cleaned data is ready now and according to your requirements can execute the analysis and make a report.



Conclusion:



Based on the SharePoint centralized database anyone can automate the complete process flow based on R. Really, this will save huge manpower and money.



If you have an automation edge kind of set up, can execute 100% automation. Yes, the bot will take care of everything, just sit and relax. 



What is NULL hypothesis?
The post SharePoint R integration and analysis appeared first on finnstats.


		

To leave a comment for the author, please follow the link and comment on their blog:  Methods – finnstats.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or  here if you don't.
The post SharePoint R integration and analysis first appeared on R-bloggers.

TL;DR

Describes the best practices for setting automated test architecture for Shiny apps. Automate and test early and often with unit tests, user interface tests, and performance tests.

Best Practices for Testing Your Shiny App

Even your best apps will break down at some point during development or during User Acceptance Tests. I can bet on this. It’s especially true when developing big, productionalized applications with the support of various team members and under a client’s deadlines pressure. It’s best to find those bugs on the early side. Automated testing can assure your product quality. Investing time and effort in automated tests brings a huge return. It may seem like a burden at the beginning, but imagine an alternative: fixing the same misbehaviour of the app for the third time e.g. when a certain button is clicked. What is worse, bugs are sometimes spotted after changes are merged to the master branch. And you have no idea which code change let the door open for the bugs, as no one checked particular functionality for a month or so. Manual testing is a solution to some extent, but I can confidently assume that that you would rather spend testing time on improving user experience rather than looking for a missing comma in the code.

How do we approach testing in Appsilon? We aim to organize our test structure according to the “pyramide” best practice:

FYI there is also an anti-pattern called the “test-cone”. Even such tests architecture in the app I would consider as good sign, after all the app is (automatically) tested – which is unfortunately often not even the case. Nevertheless switching to the “pyramid” makes your tests more reliable and effective plus less time-consuming.

No matter how extensively you are testing or planning to test your app, take this piece of advice: start your working environment with automated tests triggered before merging any pull request (check tools like CircleCI for this). Otherwise you would soon hate finding bugs caused by developers: “Aaaa, yeaaa, it’s on me, haven’t run the tests, but I thought that the change is so small and not related to anything crucial!” (I assume it goes without saying that no changes goes into ‘master’ or ‘development’ branches without proper Pull Request procedure and review).

Let’s now describe in details different types of tests:

Unit Tests

… are the simplest to implement and most low-level kind of tests. The term refers to testing the behaviour of functions based on the expected output comparison. It’s a case by case approach – hence the name. Implementing them will allow you to recognize all edge cases and understand the logic of your function better. Believe me – you will be surprised what your function can return when starting with unexpected input. This idea is pushed to the boundaries with so called Test Driven Development (TDD) approach. No matter if you’re a fan or rather skeptic at the end of the day you should have implemented the good unit tests for your functions.

How to achieve it in practice? The popular and well-known package testthat should be your weapon of choice. Add the tests folder in your source code. Inside it, add another folder testthat and a script testthat.R. The script’s only job will be to trigger all of your tests stored in testthat folder, in which you should define scripts for your tests (one script per functionality or single function – names should start with “test_” + some name that reflects the functionality or even just the name of the function). Start such a test script with context() – write inside some text that will help you understand what the test included is about. Now you can start writing down your tests, one by one. Every test is wrapped with test_that() function, with the text info what is exactly tested followed by test itself – commonly just calling the function with set of parameters and comparing the result with the expected output, e.g.

 result 

Continue adding tests for single function and scripts for all functions. Once it is ready, we can set the main testthat.R script. You can use there code: test_check(“yourPackageName”) for apps as packages or general test_results .

User Interface (UI) Tests

The core of those tests is to compare the actual app behaviour with what is expected to be displayed after various user actions. Usually it is done by comparing screen snapshots with the reference images. The crucial part though is to set up the architecture to automatically perform human-user like actions and taking snapshots. 

Why are User Interface (UI) tests needed? It is common that in an app development project, all of the functions are work fine, yet the app still crashes. It might be for example due to the JS code that used to do the job but suddenly stopped working as the object that it is looking for appears with a slight delay on the screen in comparison to what was there before. Or the modal ID has been changed and clicking the button does not trigger anything now. The point is this: Shiny apps are much more than R code with all of the JS, CSS, browser dependencies, and at the end of the day what is truly important is whether the users get the expected, bug-free experience.

The great folks from RStudio figured out a way to aid developers in taking snapshots. Check this article to get more information on the shinytest package. It basically allows you to record the actions in the app and select when the snapshots should be created to be checked during tests. What is important shinytest saves the snapshots as the json files describe the content. It fixes the usual problem with comparing images of recognizing small differences in colors or fonts on various browsers as an error. The image is also generated to make it easy for the human eye to check if everything is OK.

There is also an RSelenium package worth mentioning. It connects R with Selenium Webdriver API for automated web browsers. It is harder to configure than shinytest, but it does the job.

As shinytest is quite a new solution, in Appsilon we had already developed our internal architecture for tests. The solution is based on puppeteer and BackstopJS. The test scenarios are written in javascript, so it is quite easy to produce them. Plus BackstopJS has very nice looking reports. 

I guess the best strategy would be to start with shinytest and if there are some problems with using it, switch to some other more general solution for web applications.

Performance Tests

Yes, Shiny applications can scale. They just need the appropriate architecture. Check our case study and architecture description blog posts to learn how we are building large scale apps. As a general rule, you should always check how your app is performing in extreme usage conditions. The source code should be profiled and optimised. The application’s heavy usage can be tested with RStudio’s recent package shinyloadtest. It will help you estimate how many users your application can support and where the bottlenecks are located. It is achieved by recording the “typical” user session and then replaying it in parallel on the huge scale.

So, please test. Test automatically, early and often.

Smash down all the bugs before they become big, strong and dangerous insects!

Follow Appsilon Data Science on Social Media

Follow @Appsilon on Twitter!
Follow us on LinkedIn!
Don’t forget to sign up for our newsletter.
And try out our R Shiny open source packages!

Article Super Solutions for Shiny Architecture #5 of 5: Automated Tests comes from Appsilon Data Science | End­ to­ End Data Science Solutions.

Shiny is one of the best ways to build interactive documents, dashboards, and data science applications. But advancing your skills with Shiny does not come without challenges.

Shiny developers often have a stronger background in applied statistics than in areas useful for optimizing an application, like programming, web development, and user-interface design. Though there are many packages and tools that make developing advanced Shiny apps easier, new developers may not know these tools exist or how to find them. And Shiny developers are also often siloed. Though the Shiny developer community is huge, there is rarely someone sitting next to you to sound out ideas about your app.

With these challenges in mind, the RStudio Community has partnered with Eric Nantz of the R-Podcast to create the Shiny Developer Series.

What is the Shiny Developer Series?

Our goal is to:

• Review great tools that serve Shiny developers
• Meet the people behind these tools and learn from their experiences
• Foster the Shiny community.

Each episode of the series includes:

• A webinar, where Eric hosts a live interview with the author of a tool or package that helps make Shiny developers’ lives a bit easier.
• An open Q&A and follow-up discussion on community.rstudio.com/c/shiny
• Recorded Live Demos – when it makes sense, Eric and/or his guest will record a demo of the tools they talked about.

Register now for Shiny Developer Series updates and scheduling reminders

Past episodes

We have already had three great episodes!

Winston Chang on Shiny’s Development History and Future

In Episode 1, Winston Chang talked about the key events that triggered RStudio’s efforts to make Shiny a production-ready framework, how principles of software design are invaluable for creating complex applications, and exciting plans for revamping the user interface and new integrations.
Watch the episode – Show notes – Follow-up community discussion

Colin Fay on golem and Effective Shiny Development Methods.

In Episode 2, Colin Fay from ThinkR shared insights and practical advice for building production grade Shiny applications. He talked about the new golem package as the usethis for Shiny app development, why keeping the perspective of your app customers can keep you on the right development path, and much more.
Watch the episode – Show notes – Follow-up community discussion

Video demo of the golem shiny app development workflow

Mark Edmondson on googleAnalyticsR and building an R-Package Optimized for Shiny

In Episode 3 – Mark Edmondson from IIH Nordic talked about how he incorporates Shiny components such as modules with googleAnalyticsR and his other excellent packages. He dived into some of the technical challenges he had to overcome to provide a clean interface to many Google APIs, the value of open-source contributions to both his work and personal projects, and much more.
Watch the episode – Show notes – Follow-up community discussion

Upcoming Episodes

We have episodes scheduled for the rest of the year.

David Granjon on the RinteRface Collection of Production-Ready Shiny UI Packages

Episode 4 – Friday, August 9, Noon-1PM Eastern

If you’ve ever wanted to build an elegant and powerful Shiny UI that takes advantage of modern web frameworks, this episode is for you! David Granjon of the RinteRface project joins us to highlight the ways you can quickly create eye-catching dashboards, mobile-friendly views, and much more with the RinteRface suite of Shiny packages.

Nick Strayer on Novel Uses of JavaScript in Shiny Applications

Episode 5 – Friday, September 13, 11am-Noon Eastern

Shiny has paved the way for R users to build interactive applications based in javascript, all through R code. But the world of javascript can bring new possibilities for visualizations and interactivity. Nick Strayer joins us in episode 5 of the Shiny Developer Series to discuss the ways he’s been able to harness the power of javascript in his projects, such as his shinysense package.

Yang Tang on Advanced UI, the Motivation and Use Cases of shinyjqui

Episode 6 – Friday, October 25, 11am-Noon Eastern

Sometimes your Shiny app’s UI needs a little extra interactivity to give users more flexibility and highlight key interactions. For example, one user might not like the initial placement of a plot or data table and would like to move it around themselves. In episode 6 of the Shiny Developer Series, we will be joined by Yang Tang to discuss the development and capabilities of the powerful shinyjqui package that provides Shiny developers clean and intuitive wrappers to the immensely popular JQuery javascript library.

Victor Perrier & Fanny Meyer on dreamRs, and Tools to Customize the Look & Feel of Your App

Episode 7 – Friday November 8, 11-Noon Eastern time.

Have you ever wanted to ease the effort in customizing the look and feel of your Shiny app? Victor and Fanny are behind dreamRs, a large collection of R packages dedicated to Shiny developers, many of which are designed to help you make your Shiny app as professional looking as possible. They will talk about how moving beyond Shiny’s default options can improve your users’ experience.

Nathan Teetor on the approach and philosophy of yonder

Episode 8 – Friday December 6, 1-2pm Eastern

We have seen the Shiny community grow immensely with excellent packages built to extend the existing functionality provided by Shiny itself. But what would a re-imagination of Shiny’s user interface and server-side components entail? Nathan’s yonder package is built on Shiny, but gives developers an alternative framework for building applications with R. Eric will talk with Nathan about what shiny developers can learn from this approach and how he’s approached such an ambitious undertaking!

If you would benefit by being kept up to date about this material and developments in the community, please register now for Shiny Developer Series. You will receive updates and scheduling reminders, and check out the Shiny Developer Series website.

72-year-old British man Stephen Mckears began to question his sanity when he started noticing objects had been moved around in his shed overnight. And they weren’t just being randomly placed, things like clips and screws were somehow finding themselves neatly packed back into a tub as if to chastise Mr. Mckears for his untidiness. Just who was this fastidious phantom, this organized apparition?

Image credits: SWNS

Baffled but determined to solve the mystery, Mr. Mckears decided to empty out the tub and scatter its contents around. He was astonished to find that sure enough, everything was back in its place the next morning. Something was definitely afoot.

Image credits: SWNS

Enlisting the help of friend and neighbor Rodney Holbrook, they decided to set up a trail camera to catch the helpful ghost once and for all. Mr. Holbrook is a keen wildlife photographer, so he has experience in tracking down mystery visitors. What they found was adorably unexpected – a cute and determined mouse lifting objects twice its size in an effort to keep its ‘home’ clean.

Image credits: SWNS TV

“I’ve been calling him Brexit Mouse because he’s been stockpiling for Brexit,” Mr. Mckears said.

“The heaviest thing was the plastic attachment at the end of a hosepipe – and the chain of an electric drill. I didn’t know what it was at first. The kids were saying it was a ghost.”

Image credits: SWNS TV

“One day I emptied the tub out and spread the contents on the side – and the next day they were all back in again. I thought I was going mad.”

Image credits: SWNS TV

Mr. Holbrook was astonished at the mouse’s diligent behavior. “I’ve been calling him Metal Mickey but some people have been saying he’s just mouse proud,” he joked.

“I was quite amazed to see it – it is an amazing mouse.”

Image credits: SWNS TV

“Steve asked if he could use my trail camera to film whatever it was that was moving the objects in his shed.”

“The mouse was chucking things into the box – we thought it was a ghost or something at first. I thought I have to see this for myself.”

Image credits: SWNS TV

Image credits: SWNS TV

It seems that the mouse goes on-shift from around midnight to 2.30am – and has been doing its cleaning duties every night for about a month now.

Image credits: SWNS TV

Image credits: SWNS TV

Image credits: SWNS

“It was doing it for about two hours that night – he must have had to go for a sleep after it,” Mr. Mckears said.

Image credits: SWNS

“I’ve seen a mouse moving objects to make a nest but never metal objects. It’s quite amazing.”

“It’s still busy doing it now.”

Image credits: SWNS

People were amused by the cute and funny story

NOTE: There is some iframed content in this post and you can bust out of it if you want to see the document in a full browser window.

Also, apologies for some lingering GitHub links. I’m waiting for all the repos to import into to other services and haven’t had time to setup my own self-hosted public instance of any community-usable git-ish environment yet.

And So It Begins

After seeing Fira Sans in action in presentations at eRum 2018 I felt compelled to add support hrbrthemes support for it so I made a firasans extension to it that uses Fira Sans Condensed and Fira Code fonts for ggplot2 graphics.

But I rly wanted to go the extra mile and make an R Markdown theme for it, but I’m weary of both jQuery & Bootstrap, plus prefer Prism over HighlightJS. So I started work on “Prism Skeleton”, which is an R Markdown template that has most of the features you would expect and some new ones, plus used Prism and Fira Sans/Code. You can try it out on your own if you use markdowntemplates but the “production” version is likely going to eventually go into the firasans package. I uses markdowntemplates as a playground for R Markdown experiments.

The source for the iframe at the end of this document is here: https://rud.is/dl/hello-dorling.Rmd. There are some notable features (I’ll repeat a few from above):

• Fira Sans for headers and text
• Fira Code for all monospaced content (including source code)
• No jQuery
• No Bootstrap (it uses the ‘Skeleton’ CSS framework)
• No HighightJS (it uses the ‘Prism” highlighter)
• Extended YAML parameters (more on that in a bit)
• Defaults to fig.retina=2 and the use of optipng or pngquant for PNG compression (so it expects them to be installed — ref this post by Zev Ross for more info and additional image use tips)

“What’s this about ‘Dorling’?”

Oh, yes. You can read the iframe or busted out document for that bit. It’s a small package to make it easier to create Dorling cartograms based on previous work by @datagistips.

“You said something about ‘extended YAML’?”

Aye. Here’s the YAML excerpt from the Dorling Rmd:

---
title: "Hello, Dorling! (Creating Dorling Cartograms from R Spatial Objects)"
author: "boB Rudis"
navlink: "[rud.is](https://rud.is/b/)"
og:
  type: "article"
  title: "Hello, Dorling! (Creating Dorling Cartograms from R Spatial Objects)"
  url: "https://github.com/hrbrmstr/spdorling"
footer:
  - content: '[GitLab](https://gitlab.com/hrbrmstr)
'
  - content: 'This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.'
date: "`r Sys.Date()`"
output: markdowntemplates::prismskel
---

The title, author & date should be familiar fields but the author and date get some different placement since the goal is more of a flowing document than academic report.

If navlink is present (it’s not required) there will be a static bar at the top of the HTML document with a link on the right (any content, really, but a link is what’s in the example). Remove navlink and no bar will be there.

The og section is for open graph tags and you customize them how you like. Open graph tags make it easier to share posts on social media or even Slack since they’ll auto-expand various content bits.

There’s also a custom footer (exclude it if you don’t want one) that can take multiple content sub-elements.

The goal isn’t so much to give you a 100% usable R Markdown template but something you can clone and customize for your own use. Since this example shows how to use custom fonts and a different code highlighter (which meant using some custom knitr hooks), it should be easier to customize than some of the other ones in the template playground package. FWIW I plan on adapting this for a work template this week.

The other big customization is the use of Prism with a dark theme. Again, you can clone + customize this at-will but I may add config options for all Prism themes at some point (mostly if there is interest).

FIN

(Well, almost fin)

Kick the tyres on both the new template and the new package and drop suggestions here for the time being (until I get fully transitioned to a new git-hosting platform). One TODO for spdorling is to increase the point count for the circle polygons but I’m sure folks can come up with enhancement requests to the API after y’all have played with it for a while.

As noted a few times, the Rmd example with the Dorling cartograms is below.

The post 9th MilanoR meeting on November 20th: call for presentations! appeared first on MilanoR.

For this final post, I will cover some advanced topics and discuss how to use data tables within user generated functions. Once again, let’s use the Chicago crime data.

dat = fread("rows.csv")
names(dat) 
Let’s start by subseting the data. The following code takes the first 50000 rows within the dat dataset, selects four columns, creates three new columns pertaining to the data, and then removes the original date column. The output was saved as to new variable and the user can see the first few columns of the new data table using brackets or head function.
ddat = dat[1:50000, .(Date, value1, value2, value3)][, 
               c("year", "month", "day") := 
                        .(year(mdy_hms(Date)), 
                          month(mdy_hms(Date)),
                          day(mdy_hms(Date)))][,-c("Date")]

ddat[1:3]  # same as head(ddat, 3)

We can now do some intermediate calculations and suppress their output by using braces.
unique(ddat$month)
ddat[, { avg_val1 = mean(value1)
         new_val1 = mean(abs(value2-avg_val1))
         new_val2 = new_val1^2 }, by=month][order(month)]

In this simple sample case, we have taken the mean value1 for each month, subtracted it from value2, and then squared that result. The output show the final calculation in the brackets, which is the result from squaring. Note that I also ordered the results by month with the chaining process.
That’s all very nice and convenient, but what if we want to create user defined functions that essentially automate these tasks for future work. Let us start with a function for extracting each component from the date column.
ddat = dat[1:50000, .(Date, value1, value2, value3)]

add_engineered_dates 
We’ve created a new functions that takes two arguments, the data table variable name and the name of the column containing the date values. The first step is to copy the data table so that we are not directly making changes to the original data. Because the goal is to add three new columns that extract the year, month, and day, from the date column, we’ve used the lubridate package to define the date column and then extract the desired values. Furthermore, each new column has been labeled so that it distinctly represents the original date column name and the component that it contains. The final step was to remove the original date column, so we’ve set the date column to NULL. The final line prints the results back to the screen. You can recognize from the code above that the get function is used to take a variable name that represents column names and extract those columns within the function.
result = add_engineered_dates(ddat)
result


Let’s now apply the previous code with braces to find the squared difference between value2 and the mean value1 metric.
result[, { avg_val1 = mean(value1)
           new_val1 = mean(abs(value2-avg_val1))
           new_val2 = new_val1^2 }, by=.(Date_year,Date_month)][
                  order(Date_year,Date_month)][1:12]


Perfect!
I’m hoping that these three posts on the data.table package have convinced beginners to R of its beauty. The syntax may be scary at first, but this is in my opinion one of the most important packages in R that everyone should become familiar with. So pull your sleeves up and have fun.
If you have any comments or would like to cover other specific topics, feel free to  comment below. You can also contact me at mathewanalytics@gmail.com or reach me through LinkedIn
   

				
		


To leave a comment for the author, please follow the link and comment on their blog:  R – Mathew Analytics.

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...

How can I write R codes that other people can understand and use?

In the exercises below we cover some of the best practices while writing a small piece of R code or a full automated script. These are some of the practices which should be kept in mind while coding, trust me it will make your life a lot easier.

Answers to the exercises are available here.

If you obtained a different (correct) answer than those listed on the solutions page, please feel free to post your answer as a comment on that page.

Exercise 1
We want to create a numeric vector. The values of this vector should be between 1 and 10, starting from 1 with a difference of 2. Below is the code to generate a numeric vector.Make the suitable changes so that it follows standard practice for assignments.

NumVector = seq(1,10,by=2)

Exercise 2
The command below installs “car” package. Make changes in the command below so that all the packages get installed on which “car” is dependent.

install.packages("car")

Exercise 3
Make the changes in the below code so that it is easy for other users to read and it follows the standard practice while writing an if/else statement in R.

y <- 0
x <- 0

if (y == 0)
{
log(x)
} else {
y ^ x
}

Exercise 4
Update the below code so that it is easy for other users to read it.

NumVector <- seq(1,10,by=2)

if(length(NumVector) > 10 && debug)
message(“Length of the numeric vector is greater than 10”)

Exercise 5
Correct the indentation in the below function so that it is easy for you and other users to read and understand.

test<-1

if (test==1) {
print(“Hello World!”)
print(“The value of test is 1 here”)
} else{
print(“The value of test is not 1 here”);
}
print(test*test+1);

Exercise 6
Update the below code such that it first checks if the “dplyr” package is present. if it is already present, don’t install it just load the package.If the package is not present, install it and then load it.

install.packages("dplyr",dependencies = T)

Exercise 7
Change the below code so that the it doesn’t print package related information while loading the plyr package.

library(plyr)

Exercise 8
Make the changes in the below code so that it doesn’t print warnings while calculating the correlation value between two vectors.

a <- c(1,1)
b <- c(2,3)
cor(a,b)


Exercise 9
Update the below command so that it calls the ‘rename’ function from ‘plyr’ package. The same function is present in both the packages- ‘plyr’ and ‘rename’.

rename(head(mtcars), c(mpg = "NewName"))

Exercise 10
Create a scalar vector ‘a’ with a value of 10e-02 (1/100). Below code prints the same vector in scientific format. Make changes to print in a numeric format.

a <- 1e-02
print(a)


Monday, I will be giving the closing talk of the R in Insurance Conference, in London, on Bayesian Computations for Actuaries, as to be more specific, Getting into Bayesian Wizardry… (with the eyes of a muggle actuary). The animated version of the slides (since we will spend some time on MCMC algorithm, I thought that animated graphs could be more informative) can be downloaded from here.

Those slides are based on the chapter writen with Ben Escoto for the Computational Actuarial Science with R book, and some previous work.

J.J. Abrams is being careful not to give too much away about the upcoming Star Wars Episode VII, but there are a few things we can assume just by virtue of the fact that it’s a Star Wars movie. For example, it’s safe to say it won’t look anything like this delightful Star Wars musical parody.

Which is a real shame, because Star Wars: The Musical demonstrates that Disney tunes and lightsaber battles actually go together like peanut butter and bananas: They may not be the most intuitive combination, but they’re a damn good one. After the jump, watch Luke, Han, Leia, Darth Vader, Obi-Wan Kenobi, and more sing their way through the plot of Star Wars Episode IV: A New Hope.

Jeffrey Gee Chin directed the fan video, and George Shaw wrote the music and lyrics.

Of course Darth Vader gets to sing “When You Wish Upon a Star” about his beloved Death Star.

Obviously, there’s no way in hell Abrams will make a Disney-style Star Wars musical, even if the studio does have all the appropriate rights. But there’s a weird part of me that kinda-sorta wishes they would. If nothing else, a Star Wars musical would be a daring change of pace from all the self-serious, CG-heavy action blockbusters clogging the box office these days. 

On the bright side, there’s actually a chance that Episode VII will feature some musical performances. Musicians The Healer Twins and Safe Smokingroove have confirmed that they’ll be in the film, which makes us wonder if we’re headed back to the Mos Eisley cantina or a similar watering hole.

For more Star Wars: The Musical, visit the official website, which has tons of making-of videos and behind-the-scenes snaps. The team behind this video also promises to give the Disney treatment to Empire Strikes Back and Return of the Jedi in the future.

The post LOL: Star Wars Musical Parody Gives ‘A New Hope’ the Disney Treatment appeared first on /Film.

Guest post by Jake Russ

For a recent project I needed to make a simple sum calculation on a rather large data frame (0.8 GB, 4+ million rows, and ~80,000 groups). As an avid user of Hadley Wickham’s packages, my first thought was to use plyr. However, the job took plyr roughly 13 hours to complete.

plyr is extremely efficient and user friendly for most problems, so it was clear to me that I was using it for something it wasn’t meant to do, but I didn’t know of any alternative screwdrivers to use.

I asked for some help on the manipulator Google group , and their feedback led me to data.table and dplyr, a new, and still in progress, package project by Hadley.

What follows is a speed comparison of these three packages incorporating all the feedback from the manipulator folks. They found it informative, so Tal asked me to write it up as a reproducible example.

Let’s start by making a data frame which fits my description above, but make it reproducible:

set.seed(42)
 
types <- c("A", "B", "C", "D", "E", "F")
 
obs <- 4e+07
 
one <- data.frame(id = as.factor(seq(from = 1, to = 80000, by = 1)), percent = round(runif(obs, 
    min = 0, max = 1), digits = 2), type = as.factor(sample(types, obs, replace = TRUE)))
 
print(object.size(one), units = "GB")

## 0.6 Gb

summary(one)

##        id              percent     type       
##  1      :     500   Min.   :0.00   A:6672132  
##  2      :     500   1st Qu.:0.25   B:6663570  
##  3      :     500   Median :0.50   C:6668009  
##  4      :     500   Mean   :0.50   D:6668684  
##  5      :     500   3rd Qu.:0.75   E:6660437  
##  6      :     500   Max.   :1.00   F:6667168  
##  (Other):39997000

I’ll start the testing with plyr, using ddply, but I’ll also show the difference between subsetting a data frame from within a ddply call and doing the subset first from outside the call. Then I offer a third way to use plyr‘s count function to achieve the same result.

library(plyr)
 
## Test 1 (plyr): Use ddply and subset one with [ ] style indexing from
## within the ddply call.
 
typeSubset <- c("A", "C", "E")
 
system.time(test1 <- ddply(one[one$type %in% typeSubset, ], .(id), summarise, 
    percent_total = sum(percent)))

##    user  system elapsed 
##  104.51   21.23  125.81

## Test 2 (plyr):, Use ddply but subset one outside of the ddply call
 
two <- subset(one, type %in% typeSubset)
 
system.time(test2 <- ddply(two, .(id), summarise, percent_total = sum(percent)))

##    user  system elapsed 
##  101.20   46.14  147.64

## Test 3 (plyr): For a simple sum, an alternative is to use plyr's count
## function
 
system.time(test3 <- count(two, "id", "percent"))

##    user  system elapsed 
##    5.90    0.22    6.12

Doing the subset outside of the ddply call did speed things up, but not as much I as orinially thought it would. For my particular project, doing the subset outside of the ddply call reduced the run time to 12 hours. So largely this is still a “wrong tool” problem, rather than a “when to subset” problem.

Next, I’ll try data.table and for this test and the dplyr one below I’ll operate on the data frame which has been pre-subset:

library(data.table)
 
## Test 4 (data.table): Speed test for package data.table
 
## Define the data table
three <- data.table(two, key = c("id"))
 
tables()  # check that the key columns are correct

##      NAME        NROW  MB COLS            KEY
## [1,] three 20,000,578 310 id,percent,type id 
## Total: 310MB

## Operate on it
system.time(test4 <- three[, list(percent_total = sum(percent)), by = key(three)])

##    user  system elapsed 
##    0.17    0.01    0.19

dplyr is not currently available on CRAN but you can install it from github with:

devtools::install_github("assertthat")
devtools::install_github("dplyr")

library(dplyr)
 
## Test 5 (dplyr): Speed test for package dplyr
 
fourDf <- group_by(two, id)
 
system.time(test5 <- summarise(fourDf, percent_total = sum(percent)))

##    user  system elapsed 
##    1.49    0.03    1.52

 
sessionInfo()

## R version 3.0.1 (2013-05-16)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## 
## locale:
## [1] LC_COLLATE=English_United States.1252 
## [2] LC_CTYPE=English_United States.1252   
## [3] LC_MONETARY=English_United States.1252
## [4] LC_NUMERIC=C                          
## [5] LC_TIME=English_United States.1252    
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] dplyr_0.01       Rcpp_0.10.4      data.table_1.8.8 plyr_1.8        
## [5] knitr_1.4.1     
## 
## loaded via a namespace (and not attached):
## [1] assertthat_0.1 digest_0.6.3   evaluate_0.4.7 formatR_0.9   
## [5] stringr_0.6.2  tools_3.0.1

Both data.table and dplyr were able to reduce the problem to less than a few seconds. If you’re looking for pure speed data.table is the clear winner. However, it is my understanding that data.table‘s syntax can be frustrating, so if you’re already used to the ‘Hadley ecosystem’ of packages, dplyr is a formitable alternative, even if it is still in the early stages.

Have a nice day!

(Editor’s edition: image credit link)

It seems that the title “data science” has taken the world by storm.  It’s a title that conjures up almost mystical abilities of a person garnering information from oceans of data with ease.  It’s where a data scientist can wave his or her hand like a Jedi Knight and simply tell the data what it should be.

What is interesting about the field of data science is it’s perceived (possibly real) threat to other fields, namely statistics.  It seems to me that the two fields are distinct areas.  Though the two fields can exist separately on their own each is weak without the other.  Hilary Mason (of Bitly) shares her definition of a data scientist.  I suppose my definition differs from Hilary Mason’s data science definition.  Statisticians need to understand the science and structure of data, and data scientists need to understand statistics.  Larry Wasserman over at the Normal Deviate blog shares his thoughts on statistics and data science.  There are others blogs but these two are probably sufficient.

Data science is emerging as a field of absolutes and that is something that the general public can wrap their heads around.  It’s no wonder that statistician are feeling threatened by data scientists.  Here are two (albeit extreme) examples:

If  a statistician presents an estimate to a journalist and says “here is the point estimate of the number of people listening to a given radio station and states that the margin of error is +/- 3% with a 90% confidence interval” there is almost always a follow-up discussion about the margin of error and how the standard error was calculated (simple random, stratified, cluster) why is it a 90% confidence interval rather than a 95% confidence interval.  And then someone is bound to ask what a confidence interval is anyway?  Then extend this even further and the statistician gives the journalist a p-value?  Now there is an argument between statisticians about hypothesis testing and the terms “frequentist” and “Bayesian” start getting thrown around.

It’s no wonder that people don’t want to work with statisticians.  Not only are they confusing to the general public but the statisticians can’t even agree (even if it’s a friendly disagreement) on what is correct.  Now if we take the following data scientist example:

A data scientist looks through a small file of 50 billion records where people have listened to songs through a registration-based online radio station (e.g. Spotify, Pandora, TuneIn, etc.).  This data scientist then merges and matches the records to a handful of public data sources to give the dataset a dimensionality of 10000.  The data scientist then simply reports that there are X number of listeners in a given metro area listening for Y amount of time and produces a a great SVG graph that can be dynamically updated each week with the click of a button on a website. It is a fairly simple task and just about everyone can understand what is means.

I feel that there will always be a need for a solid foundation in statistics.  There will always exists natural variation that must be measures and accounted.  There will always be data that is so expensive that only a limited number of observations can feasibility be collected.  Or suppose that a certain set of data is so difficult to actually obtain that only a handful of observations can even be collected.  I would conjecture that a data scientist would not have a clue what to do what that data without help from someone with a background in statistics.  At the same time if a statistician was told that there is a 50 billion by 10000 dimension dataset sitting on a Hadoop cluster then I would also guess that many statisticians would be hard pressed to set the data up to analyze without consulting a data scientist.  But at the same time a data scientist would probably struggle if they were asked to take those 10000 dimensions and reduce that down to a digestible and understandable set.

Take another example of genetic sequencing.  A data scientist could work the data and discover that in one sequence there is something different.  Then a domain expert can come in and find that the mutation is in the BRCA1 gene and that the BRCA1 gene relates to breast cancer.  A statistician can then be consulted and find the risk and probability that the particular mutation will result in an increased mortality and what the probability will be that the patient will ultimately get breast cancer.

Ultimately, the way I see it the two disciplines need to come together and become one.  I see no reason why is can’t be part of the curriculum in statistics department to teach students how to work with real world data.  Those working in the data science and statistics fields need to have the statistical training while having the ability to work with data regardless of the location, format, or size.

I have a strong distaste for database connection management.  All I want to do when I want to query one of our many databases at work is to simply supply the query, and package the result into an R data.frame or data.table.

R has many great database connection tools, including but not limited to RPostgreSQL, RMySQL, RDJBC, ROracle, and RODCBC.  I set out to consolidate all of my database querying into a simple function, and I have succeeded for my purposes.  I can connect to my company’s MySQL, Postgres, and Oracle databases with ease with my fetchQuery function, defined (in conjunction with it’s dependencies) as follows.

#' Make a connection to a database
#' 
#' This function abstracts the idea of a database connection, allowing variable parameters 
#' depending on the type of database you're connecting to
#'@param config a named list of the configuration options for the database connection
#'@return a connection to the database defined in the config
#'@author Erik Gregory
makeCxn <- function(config) {
  if (class(config[['drv']]) == "character") {
    config[['drv']] <- dbDriver(config[['drv']])
  }
  do.call(dbConnect, config)
}

#' This function runs a query on a database, fetching the result if desired
#' 
#' The purpose of this function is to remove connection management from the querying process
#' @param query the query you want to make to the SQL connection you've specified
#' @param config a named list of the configuration options for the connection
#' @param n the number of rows to return, or -1 for all rows
#' @param verbose Should the queries be printed as they're made?
#' @param split Should the queries be split on semicolons, or run as a block?
#' @return A list of results if multiple queries, or a single result if one query.
#' @author Erik Gregory
fetchQuery <- function(query, config = config.gp, split = FALSE, verbose = TRUE, n = -1) {
  res <- list()
  cxn <- makeCxn(config)
  t1 <- Sys.time()
  queries <- query
  if (split == TRUE) {
    queries <- strsplit(query, ";", fixed = TRUE)[[1]] # Split the query into components
  }
  for (item in queries) {
    if(verbose) {
      cat(paste(item, '\n'))
    }
    tmp <- try(dbSendQuery(cxn, item)) # send the query
    if ('try-error' %in% class(tmp)) {
      res[[item]] <- dbGetException(cxn)
      next
    }
    type <- tolower(substring(gsub(" ", "", item), 0, 6)) # identify if select, insert, delete
    if (type == "select" | grepl("with..", type) | grepl('EXPLAI|explai', type) | !split) {
      res[[item]] <- try(fetch(tmp, n))
    }
    else {
      res[[item]] <- dbGetRowsAffected(tmp)
      cat(res[[item]])
    }
    if (verbose) {
      print(Sys.time() - t1)
      if (!is.null(dim(res))) {
        print(dim(res))
      }
    }
    dbClearResult(tmp)
  }
  dbDisconnect(cxn)
  if (length(res) == 1) {
    res <- res[[1]]
  }
  res
}

I set my default config parameter to fetchQuery to be my most commonly used connection. I define my connections in my file at ~/.Rprofile file. The effect of this is that I always have the configuration information in memory whenever I need them. An example is as follows:

config.gp <- list(
    user = "username",
    password = "password",
    dbname = "MY_DBNAME",
    host = "url_of_host",
    port = port_of_host,
    drv = "PostgreSQL"
)

config.gp.admin <- list(
    user = "username_of_admin",
    password = "password_of_admin",
    dbname = "MY_DBNAME",
    host = "url_of_host",
    port = port_of_host,
    drv = "PostgreSQL"
)
config.mysql <- list(
    user = "username",
    password = "password",
    dbname = "MY_DBNAME",
    host = "MY_HOST_IP_ADDRESS",
    drv = "MySQL"
)
config.whse <- list(
  drv = JDBC("oracle.jdbc.OracleDriver", "/usr/lib/oracle/instantclient_11_2/ojdbc5.jar"),
  user = "username",
  password = "password",
  url =  "url"
)

Notice that the drv (driver) argument can be either an actual driver, or a character string of the driver type. My reason for this is that some driver initializations require multiple parameters, while some only require a single one. This could be made elegant by using a do.call argument, as defined in makeCxn above.

It is important that the connection lists defined in the ~/.Rprofile file

1. Have the arguments you want to pass to dbConnect, named according to the value they correspond to in dbConnect
2. Have no extra arguments that you don’t want to pass to dbConnect

Some explanations behind the reasoning of the arguments and methods of fetchQuery:

• Sometimes I want to run a bunch of queries all contained in one character string to my databases. This function will either split those queries by semicolons, or run them all in one batch depending on what you ask it to do. The advantage of the former is you will have diagnostics for each of the intermediary queries (temporary table creations, table deletions or inserts, …).
• I usually want to see how my query is doing as it’s running, so I provide a verbosity option.
• fetchQuery attempts to auto-detect whether you’re doing an insert or deletion or selection, and returns a result appropriate to the operation. This algorithm is simple, crude string-matching at this point, and i’d be happy to see an improvement. It hasn’t been a problem for me yet since I am very consistent in my sql syntax.

So, whenever I want to run an oracle query i’ll run something like:

res <- fetchQuery("SELECT * FROM table_name", config.whse)

or if I want to run a query as an admin against our Postgres database

res <- fetchQuery("SELECT * FROM table_name limit 10", config.gp.admin)

The connection fortunately gets closed no matter if the query is in error or not, which is really nice for me and my company’s DBAs (R will limit the number of active database connections you can have open, so it is important to close them).

The most influential people strive for genuine buy in and commitment — they don't rely on compliance techniques that only secure short-term persuasion. That was our conclusion after interviewing over 100 highly respected influences across many different industries and organizations for our recent book.

These high-impact influencers follow a pattern of four steps that all of us can put into action. In earlier pieces we covered Step 1: Go for great outcomes and Step 2: Listen past your blind spots. Later we'll cover Step 4: When you've done enough... do more. Here we cover Step 3: Engage others in "their there."

To understand why this step is so important, imagine that you're at one end of a shopping mall — say, the northeast corner, by a cafe. Next, imagine that a friend of yours is at the opposite end of the mall, next to a toy store. And imagine that you're telling that person how to get to where you are.

Now, picture yourself saying, "To get to where I am, start in the northeast corner by a cafe." That doesn't make sense, does it? Because that's where you are, not where the other person is.

Yet that's how we often try to convince others — on our terms, from our assumptions, and based on our experiences. We present our case from our point of view. There's a communication chasm between us and them, but we're acting as if they're already on our side of the gap.

Like in the shopping mall example, we make a mistake by starting with how we see things ("our here"). To help the other person move, we need to start with how they see things ("their there").

For real influence we need to go from our here to their there to engage others in three specific ways:

1. Situational Awareness: Show that You Get "It." Show that you understand the opportunities and challenges your conversational counterpart is facing. Offer ideas that work in the person's there. When you've grasped their reality in a way that rings true, you'll hear comments like "You really get it!" or "You actually understand what I'm dealing with here."
2. Personal Awareness: You Get "Them." Show that you understand his or her strengths, weaknesses, goals, hopes, priorities, needs, limitations, fears, and concerns. In addition, you demonstrate that you're willing to connect with them on a personal level. When you do this right, you'll hear people say things like "You really get me!" or "You actually understand where I'm coming from on this."
3. Solution Awareness: You Get Their Path to Progress. Show people a positive path that enables them to make progress on their own terms. Give them options and alternatives that empower them. Based on your understanding of their situation and what's at stake for them personally, offer possibilities for making things better — and help them think more clearly, feel better, and act smarter. When you succeed, you'll hear comments like, "That could really work!" or "I see how that would help me."

One of our favorite examples involves Mike Critelli, former CEO of the extraordinarily successful company, Pitney Bowes. Mike was one of the highly prestigious Good to Great CEOs featured in the seminal book by Jim Collins on how the most successful businesses achieve their results.

One of Mike's many strengths is the ability to engage his team on their terms to achieve high levels of performance and motivation. When we asked him about this, he said, "Very often what motivates people are the little gestures, and a leader needs to listen for those. It's about picking up on other things that are most meaningful to people."

For example, one employee had a passing conversation with Mike about the challenges of adopting a child, pointing out that Pitney Bowes had an inadequate adoption benefit. A few weeks after that, he and his wife received a letter from Mike congratulating them on their new child — along with a check for the amount of the new adoption benefit the company had just started offering.

When he retired, the Pitney Bowes employees put together a video in which they expressed their appreciation for his positive influence over the years. They all talk about ways that Mike "got" them — personal connections and actions that have accumulated over time into a reputation that attracted great people to the organization and motivated them to stay.

It's a moving set of testimonials, and it's telling about Critelli's ability to "get" people on their own terms — to go to their there — that they openly express their appreciation permanently captured on video for open public viewing.

Remember, they did this after he was no longer in power.

Like Mike Critelli does, when you practice all three of these ways of "getting" others — situational, personal, and solution-oriented — you understand who people are, what they're facing, and what they need in order to move forward. This is a powerful way to achieve great results while strengthening your relationships.

When you're trying to influence, don't start by trying to pull others into your here. Instead, go to their there by to asking yourself:

• Am I getting who this person is?
• Am I getting this person's situation?
• Am I offering options and alternatives that will help this person move forward?
• Does this person get that I get it?