StoreException: could not connect to database - After upgrade to 3.8.0

[LGJLI]

Issue description

If you found a bug or want to have a new feature implemented, please take your time and look whether this issue has been discussed in another issue or on our mailing list already.

SCM-Manager is open source. If you have any knowledge of programming, maybe you can help. We would love to support you.

Bug Report

Describe how to reproduce the bug

When upgrading using apt repos from 3.7.3 to 3.8.0 the server hangs with the following error:

om.google.inject.ProvisionException: Unable to provision, see the following errors:

1) [Guice/ErrorInjectingConstructor]: StoreException: could not connect to database
  at SQLiteQueryableStoreFactory.<init>(SQLiteQueryableStoreFactory.java:64)
  at SQLiteQueryableStoreFactory.class(SQLiteQueryableStoreFactory.java:47)
  while locating SQLiteQueryableStoreFactory
  at RepositoryQueryableStoreExporter.<init>(RepositoryQueryableStoreExporter.java:53)
      \_ for 2nd parameter
  at RemainingQueryableStoreImporter.<init>(RemainingQueryableStoreImporter.java:45)
      \_ for 2nd parameter
  while locating RemainingQueryableStoreImporter

Learn more:
  https://github.com/google/guice/wiki/ERROR_INJECTING_CONSTRUCTOR

1 error

Revering back to 3.7.3 fixes the issue

Describe your SCM-Manager setup

Ubuntu 22.04 server - running SCM-manager 3.8.0

Which operating system, browser and versions do you use?

Firefox browser on Windows.

Log of the bug

  com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) [Guice/ErrorInjectingConstructor]: StoreException: could not connect to database
  at SQLiteQueryableStoreFactory.<init>(SQLiteQueryableStoreFactory.java:64)
  at SQLiteQueryableStoreFactory.class(SQLiteQueryableStoreFactory.java:47)
  while locating SQLiteQueryableStoreFactory
  at RepositoryQueryableStoreExporter.<init>(RepositoryQueryableStoreExporter.java:53)
      \_ for 2nd parameter
  at RemainingQueryableStoreImporter.<init>(RemainingQueryableStoreImporter.java:45)
      \_ for 2nd parameter
  while locating RemainingQueryableStoreImporter

Learn more:
  https://github.com/google/guice/wiki/ERROR_INJECTING_CONSTRUCTOR

1 error

======================
Full classname legend:
======================
RemainingQueryableStoreImporter:  "sonia.scm.importexport.RemainingQueryableStoreImporter"
RepositoryQueryableStoreExporter: "sonia.scm.importexport.RepositoryQueryableStoreExporter"
SQLiteQueryableStoreFactory:      "sonia.scm.store.sqlite.SQLiteQueryableStoreFactory"
StoreException:                   "sonia.scm.store.StoreException"
========================
End of classname legend:
========================

	at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:251)
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1151)
	at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1186)
	at sonia.scm.lifecycle.BootstrapContextListener.processUpdates(BootstrapContextListener.java:176)
	at sonia.scm.lifecycle.BootstrapContextListener.startEitherMigrationOrApplication(BootstrapContextListener.java:134)
	at sonia.scm.lifecycle.BootstrapContextListener.createMigrationOrNormalModuleProvider(BootstrapContextListener.java:125)
	at sonia.scm.lifecycle.BootstrapContextListener.createStageTwoInjector(BootstrapContextListener.java:118)
	at sonia.scm.lifecycle.BootstrapContextListener.getInjector(BootstrapContextListener.java:90)
	at com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:45)
	at sonia.scm.lifecycle.BootstrapContextListener.contextInitialized(BootstrapContextListener.java:68)
	at sonia.scm.lifecycle.BootstrapContextFilter.initializeContext(BootstrapContextFilter.java:57)
	at sonia.scm.lifecycle.BootstrapContextFilter.init(BootstrapContextFilter.java:48)
	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
	at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$2(ServletHandler.java:725)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:749)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1305)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:902)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:533)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
	at org.eclipse.jetty.server.Server.start(Server.java:470)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
	at org.eclipse.jetty.server.Server.doStart(Server.java:415)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at sonia.scm.server.ScmServer.init(ScmServer.java:65)
	at sonia.scm.server.ScmServer.run(ScmServer.java:36)
	at sonia.scm.server.ScmServerDaemon.main(ScmServerDaemon.java:34)
Caused by: sonia.scm.store.StoreException: could not connect to database
	at sonia.scm.store.sqlite.SQLiteQueryableStoreFactory.openDefaultConnection(SQLiteQueryableStoreFactory.java:114)
	at sonia.scm.store.sqlite.SQLiteQueryableStoreFactory.<init>(SQLiteQueryableStoreFactory.java:91)
	at sonia.scm.store.sqlite.SQLiteQueryableStoreFactory.<init>(SQLiteQueryableStoreFactory.java:64)
	at sonia.scm.store.sqlite.SQLiteQueryableStoreFactory$$FastClassByGuice$$1350ea2.GUICE$TRAMPOLINE(<generated>)
	at sonia.scm.store.sqlite.SQLiteQueryableStoreFactory$$FastClassByGuice$$1350ea2.apply(<generated>)
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:60)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:40)
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:60)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:40)
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:60)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1148)
	... 35 more
Caused by: org.sqlite.SQLiteException: [SQLITE_BUSY] The database file is locked (database is locked)
	at org.sqlite.core.DB.newSQLException(DB.java:1179)
	at org.sqlite.core.DB.newSQLException(DB.java:1190)
	at org.sqlite.core.DB.execute(DB.java:985)
	at org.sqlite.core.CoreStatement.exec(CoreStatement.java:91)
	at org.sqlite.jdbc3.JDBC3Statement.lambda$execute$0(JDBC3Statement.java:56)
	at org.sqlite.jdbc3.JDBC3Statement.withConnectionTimeout(JDBC3Statement.java:458)
	at org.sqlite.jdbc3.JDBC3Statement.execute(JDBC3Statement.java:44)
	at org.sqlite.SQLiteConfig.apply(SQLiteConfig.java:220)
	at org.sqlite.SQLiteConnection.<init>(SQLiteConnection.java:70)
	at org.sqlite.jdbc3.JDBC3Connection.<init>(JDBC3Connection.java:28)
	at org.sqlite.jdbc4.JDBC4Connection.<init>(JDBC4Connection.java:19)
	at org.sqlite.JDBC.createConnection(JDBC.java:106)
	at org.sqlite.SQLiteDataSource.getConnection(SQLiteDataSource.java:473)
	at org.sqlite.SQLiteDataSource.getConnection(SQLiteDataSource.java:465)
	at sonia.scm.store.sqlite.SQLiteQueryableStoreFactory.openDefaultConnection(SQLiteQueryableStoreFactory.java:110)
	... 58 more

Further information like screenshots

[pfeuffer]

Hi @LGJLI , thanks for this report!

Did you do a rollback on your data, too?
If not, can you do us a favour and take a look at the SCM home directory? There should be a file scm.db and maybe some more with further extensions (like -shm or -wal). If you find these files, can you take a closer look at the permissions for these files?

Thanks for your help!

[LGJLI]

Only scm.db is present. I tried deleting it before upgrading (after publising the issue), but to no avail.

What filesystem permissions should the scm.db file have ?

This is the current permissions

 lg@vcs:/mnt/scm-home$ getfacl scm.db
# file: scm.db
# owner: root
# group: root
user::rwx
group::r-x
other::r-x

[pfeuffer]

Thanks for the quick response. This is interesting. The file should have the user (and probably the group) of the process running SCM-Manager (so just like all the other files in this directory). I assume that SCM-Manager is not running a root?

I'm wondering, why the file was created with root. Did you start the server once in another way than the other times?

[LGJLI]

I installed SCM-manager using the installation guide here (v.3.1.x)
Could I have misconfigured anything to make it run as root ?

systemctl show -pUser,UID scm-server
UID=998
User=scm

I only start the scm-server using systemctl when upgrading.
Not sure if I started the server as root back when I first installed it...

Just want to mention that I have placed scm-home on a network share.

[tidiegeler]

Hello @LGJLI, thank you for your feedback. We are going to take a look at this issue at a later time.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[zT-1337]

Hello @LGJLI,

after several attempts we could not reproduce this issue. We would advise to change the owner and group of the scm.db file to match the user/group of the other files within the scm-home directory. Afterwards the application should be able to run. If you find any other issues, feel free to reach out to us.

[ethanpeterson]

I ran into this same issue when upgrading from v3.7.5 to both v3.8.0 and v3.9.0. The error immediately occurred when starting the scm server after the upgrade and I noticed the scm.db was 0 bytes.
My workaround was to take a backup of the home folder (before the upgrade), uninstall scm server, then run a fresh install. After a successful install, I stopped the scm server, copied the backup into the home folder (not overwriting any existing files) and restarted the scm server. So far so good.

[pfeuffer]

Thanks a lot for mentioning this here, @ethanpeterson ! Can you tell us, what OS you are using and how you installed SCM-Manager (deb Package, native packate, ...)?