Use KKT conditions to speed up mode search

[msuchard]

Hi @schuemie and @pbr6cornell ,

If you can generate a dataset with 100,000 or more covariates and many, many rows, I just implemented (in a15bb66) a mode search strategy that should be much faster than before. Please let me know your mileage, so I can tinker a bit with performance. The R commands are:

slowFit <- fitCyclopsModel(massiveCyclopsData,
                           forceNewObject = TRUE, # Cold start for fair comparison
                           prior = createPrior("laplace"))

fastFit <- fitCyclopsModel(massiveCyclopsData,
                          forceNewObject = TRUE, # Cold start for fair comparison
                          prior = createPrior("laplace"),
                          control = createControl(noiseLevel = "quiet",
                                                  useKKTSwindle = TRUE,
                                                  tuneSwindle = 10)) # Maybe try 50, 100 as well

[schuemie]

Did a first assessment with a 'medium-sized' set (24k covariates, 450k rows), which was not a big success:
With KKTSwindle: 8.5 hours
Without KKT Swindle: 16 minutes
Will try with a much bigger set now.

[msuchard]

Well, that certainly seems disappointing, and counter to the small examples with which I played; using the swindle never took longer.

Can you share the data files? Or provide R commands that I can run against cdm_sim4 to generate something similar?

[msuchard]

The following shows an approximately 2-fold speed-up:

    seed <- 666
    tolerance <- 1E-4

    data <- simulateCyclopsData(nstrata = 1,
                                nrows = 10000,
                                ncovars = 20000,
                                zeroEffectSizeProp = 0.99,
                                model = "logistic")

    cyclopsData <- convertToCyclopsData(data$outcomes,
                                        data$covariates,
                                        modelType = "lr",
                                        addIntercept = TRUE) 

    slowFit <- fitCyclopsModel(cyclopsData,
                               forceNewObject = TRUE, # Cold start for fair comparison
                               prior = createPrior("laplace", variance = 0.01, exclude = c(0)))


    fastFit <- fitCyclopsModel(cyclopsData,
                               forceNewObject = TRUE, # Cold start for fair comparison
                               prior = createPrior("laplace", variance = 0.01, exclude = c(0)),
                               control = createControl(noiseLevel = "silent",
                                                       useKKTSwindle = TRUE,
                                                       tuneSwindle = 100))    

    slowFit$timeFit
    fastFit$timeFit

    slowFit$timeLog # An apparent inefficiency in moving coefficient estimates back into R
    fastFit$timeLog #

    expect_equal(coef(slowFit), coef(fastFit), tolerance = tolerance)

[msuchard]

Am still wondering about this ....