Jenkins: Retrieving submodules with Git

Currently I've been stuck on an issue trying to retrieve the submodules of a repository from within Jenkins. My configuration is fine and I can pull repositories without any submodules just fine.

I can also pull the main components of a repo with submodules (both with authentication in the repository-name as with SSH). The problem only arises when I have to pull the submodule-components. I'm running the latest version of Jenkins and I've added a part at the bottom which is for "Advanced sub-modules behaviours". I selected "Recursively update submodules" here and ran the build several times to no avail.

When I try adding an extra build step at the bottom with shell-commands, the updating of the repositories doesn't work either. When I try these commands outside of jenkins in my terminal, this works just fine. The issue I always get in Jenkins is:

FATAL: Command "git submodule update" returned status code 1:

stdout: 

stderr: Cloning into 'thisismysubmodule'...
fatal: Authentication failed for 'https://git.thisismyrepo.com/scm/ap/thisismysubmodule.git/'

I've found this issue: https://issues.jenkins-ci.org/browse/JENKINS-20941 but I can't use the suggested solution at the bottom due to security concerns. Does anyone here have any experience with this problem or a possible solution?

Answers


Here is a workaround using SSH agent forwarding. It worked fine for me.

  1. First, edit <jenkins_home>/.ssh/config and set ForwardAgent yes
  2. Then, install SSH Agent plugin for Jenkins.
  3. Then, in project configuration, reset the Git credentials.
  4. Finally, in project configuration, set the SSH Agent credential.


Beta versions of the git-client and git-plug-in modules have been released now to solve this issue. To quote the JIRA issue.

The git client plugin 2.0.0-beta1 has been released to the experimental update center. It includes git submodule authentication, JGit 4.3, and requires JDK 7. It requires at least Jenkins 1.625 (the first version to mandate JDK 7)

Using the above there is an option under the Additional Behaviours section called:

Use credentials from default remote of parent repository

This solved my issue when using Jenkins 2.11 and the beta versions of the plug-in running a Windows server and slave. I have not checked other build machines. Also you must use the same authentication method, if using http you must use that for submodules as well, if use SSH you must use this for the submodules, trying to mix methods will not work correctly.

-- UPDATE -- Beta versions are not required anymore, please see the following pages: Git Client Plugin Git Plugin


One solution would be to declare in the global git config file a netrc credential help, which would provide the necessary credentials for any http query coming from git.

git config --global credential.helper "netrc -f C:/path/to/_netrc.gpg -v"

(make sure to use the same account as the one used for running Jenkins)

I use an encrypted netrc file, but you can start some test with an un-encrypted one.


Considering I have tried pretty much all available options to get this working (SSH, .netrc, hardcoded credentials,...) the only option was the one which was mentioned on the bottom of the JENKINS-20941 issue by 'andreg':

Yeah, this issue is a real pain for us too. The only way we could get it working for our Stash/Jenkins setup was to create a read-only user and to hard code this user's credentials in the reference to the submodule. Although bad practise, all users working on the git repo already have at least read-only access, so we didn't feel it was too much of a security concern. e.g. in the parent repo .gitmodules file:

[submodule "shared-library"] path = shared-library url = https://username:password@stash.yourcompany.com/scm/project/shared-library.git

and then the Jenkins job has "Recursively update submodules" selected.


I actually just moved it to a shell command and in the shell command I told it which credentials helper to use, since I'm on windows it was wincred:

git config --global credential.helper wincred
git submodule init 
git submodule sync 
git submodule update --init --recursive

I managed to get this working by simply adding a .netrc file with the credentials on linux. Not the most secure solution but if you need to get it working quickly it will get you going.


This solution worked for me:

  1. Make sure ssh credentials are the same for the parent repo and the submodule repo.
  2. Setup credentials for the parent repo in Jenkins. (Source Code Management (choose "Git") > Repositories)
  3. Setup your .gitmodule file so that it uses the ssh credentials from the parent repo:

    [submodule "foo/repository"]
       path = foo/repository
       url = ssh://jenkins_build_monkey@domain.com/repository
    
  4. There is a small caveat to this solution. Anytime that someone clones the parent repo, they will need to sync the url to one that they have access to. The first time a user initializes the submodule, they need to first edit the .gitmodules file and change the url:

    [submodule "foo/repository"]
       path = foo/repository
       url = ssh://username@domain.com/repository
    

    Then, in the terminal:

    git submodule sync
    git submodule init repository
    git submodule update --remote repository
    

    And then change the url back to the jenkins build url. Unless you sync again, the submodule will use the url associated with the user.

In September 2016, Jenkins is planning to release a new feature that will allow submodules to share the credentials of the parent repository. Then a HTTP url can be used in .gitmodule instead of ssh.


Need Your Help

Change color of the drop down arrow of Spinner in XML

android android-layout colors android-spinner android-color

As I wrote in my question, I want to change the color of the drop down arrow (the default arrow, not a custom arrow or something like that) of a Spinner in XML, but the problem is that I couldn't ...

Rendering React components with promises inside the render method

javascript reactjs q

I have a component which gets a collection of items as props and maps them to a collection of components which are rendered as children of a parent component. We use images stored in WebSQL as byte